We've just enabled submissions for Flight Quest 2. To make your first submission, download sampleSubmission.csv from the data page and upload it under "Make a Submission". The submission should take about 10s to process, or a bit longer if another FQ2 submission is in the queue ahead of you. The code to create this submission is available on Github.

This is an alpha release for submissions on this competition. If you only like participating in competitions where the data and evaluation metric are more stable and less likely to be updated, come back in a couple weeks. For this competition, we will likely be updating the evaluation metric over the next one to two weeks. As it is being updated, your leaderboard scores may change or submissions made before the update may be discarded.

However, we wanted to go ahead and let you get used to the submission process and start exploring the competition while load-testing our system. You will be able to tackle the problem in full force when we release the simulator code and the full test dataset.

When you make a submission, it should contain a list of waypoints for the plane to follow along its flight. These waypoints should take it to within 75 standard miles of the destination airport at an altitude of 18,000 feet. The sample submission has two waypoints for most flights: one that's 150 miles from the airport at 38,000 feet and a second that's directly over the airport at 18,000 feet (which will cause the plane to descend and then trigger the landing model once it's within 75 miles of the airport at an altitude of at most 18,000 feet).

The key components of the current version of the Simulator are as follows:

• Fuel model: This is a function of the plane's current weight, airspeed and altitude. It governs how much fuel the plane burns during the cruise portion of the flight and during its changes in altitude. This fuel model is for a single model, medium-range plane type.
• Wind information: The simulator uses NOAA's Rapid Refresh product to estimate the current headwind/tailwind the plane is experiencing. Since the instructions have the plane flying at a specific airpseed, this impacts the observed groundspeed of the plane, the time it will take for a plane to complete the route, and the corresponding fuel burn.
• Landing model: When the plane is at an altitude below 18,000 ft and within 75 miles of the destination airport, any subsequent waypoints are ignored. A basic predictive model that takes into account airport ground conditions and traffic is used to predict the remaining duration of the flight along with the remaining fuel usage.
• Altitude corrections: Pilots must fly at designated flight levels to avoid collision. Westbound flights travel at even thousands of feet; Eastbound flights travel at odd thousands of feet. Each waypoint altitude is rounded to the nearest flight level that is less than the aircraft's ceiling. For instance: Given an aircraft with a 41,000 ft ceiling travelling westbound that has an instruction to reach 43,284 ft, the target altitude will be changed to 40,000 ft.
• Cost model: A relative cost for the flight is calculated as a function of the fuel burned and the amount of time delayed. The parameters for the cost function and number of passengers are selected randomly from a pre-specified range for each flight. These do not correspond to any real world costs for the flight.

This is by far the most complex competition we've ever hosted, which means there is a large surface area for potential bugs and issues in both the data and the Simulator. Please report any issues that you find on the competitions forums in a constructive manner. Remember, as George Box said, "All models are wrong, some models are useful." The current version of the Simulator is by no means a complete reflection of reality nor does it account for all the considerations that need to be made when determining a flight route.

The Simulator is not currently taking into account the following constraints:

• Limiting the flight to specific airways or routes
• Static no-fly zones due to special use airspace restrictions
• Moving no-fly zones due to storms and other weather
• Explicit fuel models for multiple types of aircraft (though some of these tradeoffs are accounted for by the spectrum of parameters in the cost function)
• Airport curfews
• Direction from which a plane should approach a specific airport for landing
• Turbulence

At the Milestone, the Simulator may be updated to include one or more of the above considerations or other features that are not listed above.

Thank you for the update. This competition can be quite interesting and challenging. I have 2 questions. They are rather philosophical because I want to understand the bigger picture.

1. I wonder if you took into consideration how optimal the real flights are in general by your definition. There are a 2 options:

a. Most flights are unoptimal - you need to greatly change the path 

b. Most flights are optimal - all you need to do is to make some minor corrections

To be honest I would be surprised if option a. was pervasive in real life.

2. What is the basis for calculating the time delay? Is the actual runway arrival time a lower bound for the arrival in our simulation? For example the pilot gets an information that given the current path the flight will be delayed by 10 minutes because of the traffic at the airport. Any simulation that  arrives sooner should be penalized. The good part for the pilot is that the 10 minutes slack can be utilzed to redistribute speed over the path and that's where the savings can be made. Do I understand it correctly?

Also by this question I want to understand the importance of predicting the arrival time and how this phase relates to the first phase of the competition.

As the simulator uses NOAA Rapid Refresh data, could we use it in our models as well?

If so, will you provide NOAA data for the training period or the leaderboard testing period (at least up to the cut-off time)?

A few side notes/questions:

a) in testFlights.csv, it looks like the columns StandardPassengerCount and PremiumPassengerCount should be swapped (values like 12 standard and 130 premium passengers do not look right).

b) you're providing CurrentGroundSpeed, but no current facing direction. Are we to assume the aircraft is facing towards the destination airport always?

c) How do you determine the aircraft's flight ceiling? Do you have the aircrafts database built into the simulator? If so, could we re-use this data in our models as well?

Ben,

Thanks for the post.

I think you have addressed most of my questions and concerns regarding the evaluation metric in this message and also in the update to the data set we should use.

Ben,

Where do I find basicAgent.py.  I've scanned the instructions and can't immediately find any reference to it, and Github is a big place!

'This submission, sampleSubmissions.csv, was created with basicAgent.py, which is available on Github.'

Go to the specified GIT directory and download the zipfile. Inside there (i.e. in the "src" dir) is the relevant python prog.

But where is the specified GIT directory? where exactly in the documentation is it specified?

Where it says "is available on github" at the top of this thread. Click the link.

As far as I can tell the sample submission is just a guess - nice code to tell us how massage the data but not insightful algorithm wise.

I'm still struggling to understand how we add 'intelligence' to our solution without using the simulator as an integral part of our solution. And if we can, what stops us from just testing all possible flight paths?

Am I obviously clearly missing some vital understanding here and would appreciate some clarification.

It might tell you something. I assume it already gives the best cost when all aircraft are assigned the same kinds of waypoints. :)

It would be interesting to know whether the simulator assumes a planar, spherical or elliptical planet.

Also, is it possible to add an explanation of variables for the two new data sets to the Data Page?

We have updated the Flight Simulator this morning to fix a bug that treated Eastbound flights as Westbound, and therefore giving them wrong flight levels. 

After applying the fix, we noticed that our sampleSubmission.csv file is not landing all the planes.  The fault lies with the final waypoint altitude of 18,000 ft - the max altitude needed to reach the arrival zone.  However, when Eastbound flights have an instruction for 18,000, InstructionCorrector.fs will round that to the nearest flight level 19,000 ft - missing the arrival zone.

Anyone that has only submitted the sampleSubmission.csv will see their score skyrocket to ~59,000 (from frequent non-arrival penalties) once we re-score all the submissions.  You're welcome to submit our updated sampleSubmissionRev2.csv which has all final altitudes set to 17,000 ft. 

We'll update the Simulation code shortly after re-scoring all the submissions today.

Update: Submissions have been re-scored and the simulator code has been updated.

Am I the last or the first to find out that you don't have to submit all 14,989 flights (testFlightsRev3.csv) to the evaluation process?  Is this deliberate?  If so it enables 'gaming' the system by submitting only those flights that have the best results.  This doesn't seem compatible with the competition's objectives.

The feature has been mentioned.

I don't think gaming is possible. From what I understood a while back any flight with no waypoints doesn't arrive and you're hit with the penalties. Seems fair. :)

if the leaderboard simulator behave as the released simulator, then if you don't have waypoints for a flight, that flight will have a cost of the respective NonarrivalPenalty.

Currently the NonarrivalPenalty is $100k for all the flights, which is well above the most expensive flight I've come across in the OneDaySimulator (most expensive I've come across is below $50k). Hence, not submitting the most expensive flights will give you a much higher penalty anyhow!

Thanks Alessandro.  I should have realised your explanation.  However I'm sure all my flights finish within 75 miles so I don't know why my submission(s) result in such terrible scores.