Can some of you explain to me, what's about the model submission deadline? What exactly must be submitted up to this date? And what can be changed/done on the model between the model submission deadline and the final deadline?

The model submission deadline is the date by which you need to "lock down" your code, stop development, and upload the code to our website on the Submissions page. No further changes can be made to the model after the model submission deadline.

We will release the Final Evaluation Data Set a few weeks later and there will be a new leaderboard. You need to use your code to create a set of waypoints for the flights in the new test flights file and upload that submission. 

The Final Evaluation Set leaderboard, and a review of the code, will determine the winners of Flight Quest 2.

Just to be sure: it is only the code that is fixed? Because I change things in my input files (like I exctract from date the day and make a new colum with day only, which is then read into my code or I never bothered skipping line 1 in my code when reading, I just removed the first line in the input files, etc.).

It is also important for me to know whether the new test flights will be again 14 days (some loops are based on that)? And if we get again the at cut-off weather-files?

A code freeze is simple enough when everyone is on the same team, but I wonder how this is going to play out in a contest. It seems awfully subjective and unenforceable.

Is it an absolute freeze, or are "reasonable" bug fixes and/or adjustments allowed? On a case by case basis?

My program reads in tables from text files. May I modify the tables after the deadline or is that "code?" How about modifying "a few" parameters via config files?

Continuing development after the code freeze could bring advantages that are unrelated to looking up historical data: e.g. tuning parameters to match the final data set, or tweaking code for general performance improvement. How is Kaggle going to verify that the model submitted is exactly the same as the one that produced the paths for the final data set? If not, how much leeway are we permitted?

My program is stochastic. Even with a fixed seed, it might not be perfectly repeatable because the simulator itself appears to run very slightly differently from one machine to the next. A more likely scenario would be that I have to interrupt my program when time runs out, or I simply fail to record something. Running it again should be close but Kaggle contests are known for tight races.

I think of what I currently have as more of a process than a model. It makes simple paths and then performs a sequence of refining steps to try to improve them using different methods. Time management is a big deal. If I truly have 12 days, I would want to set it up one way. But if the time available changed, I'd like the flexibility to adjust the number of iterations for some steps, the threshold for others... Actually, I'd like to preserve that flexibility anyway, or at least have high confidence that everyone's adhering to the same interpretation of this rule.

The code freeze is absolute. Any bug fixes will be handled on a case-by-casing basis, with the participant needing to demonstrate that it was only a bug fix instead of an improvement to their methodology. For example, say a NA value was in a column that previously had no NA values, and this caused your code to crash. Fixing this bug in order to make a submission would likely be fine. As another example, modifying or optimizing a parameter based on the test set would be considered a methodology improvement and invalidated.

Thank you. What about changing parameter settings for reasons of time management? The schedule gives 12 days for processing the final data set. For one reason or another, it might not turn out that way.

That's a gray area where the would details matter. If you're running an optimization method for a set period of time or a set number of iterations, an alternative approach could be to include a specific datetime (say two days before the final submission deadline) as the stopping condition instead.

Going through a complete list of hypothetical situations where ambiguities might arise likely isn't a productive use of time. The best guidance I can give is as follows:

• Your operating assumption should be that you're running the exact same code as you submitted to produce your final results. Any changes required to run your code will be approved on a case-by-case basis.
• Changes that are clearly and demonstrably bug fixes (such as correcting a file name / handling an unexpected data input on a single row that causes your program to crash) are likely to be allowed.
• Changes that affect the performance of your methodology (e.g. further tuning a learning parameter that had been explicitly coded into your model based on the new data) are likely to be disallowed.

A friendly reminder that the Model Submission Deadline (Saturday, January 18, 2014 11:59 PM UTC) is approaching. Model submissions have now been turned on so please upload your models by the deadline in order to continue participating in Flight Quest 2 and be eligible for the prizes. Please see the information in this thread for details about model submission.

The remaining Competition Timeline can be found here.

A really dumb question.

Exactly what it to be submitted?  An executable along with a read me file?  Or the whole source code? or what.  I'm thinking I saw this somewhere on the web site - but now I can't find it so if this information is around here please point me to it.

Robert Ramey

When you submit an Entry, you also will be able to upload your model to the Website. Your final model must contain all code and parameter settings necessary to evaluate your models on new data, and include a README file with instructions on how to do so. The purpose of this is to ensure this Competition is fair and no answers were looked up for the Final Evaluation Set period. This model submission is required to be eligible to win a prize. This final model will be used solely for the purpose of verifying eligibility for prizes. You may upload either the archive itself or a SHA-256 hash of the archive.
Kaggle will check the final submissions from the Entrants at the top of the Final Evaluation Leaderboard to make sure that Entrant’s final submission was created using that Entrant’s final model submission. If a potential winner did not use the final model codebase to create the final submission Entry, that potential winner will be disqualified and Kaggle will evaluate the final submission of the next Entrant on the Leaderboard as a potential winner.

For an entry to be eligible to win, any open source or third party code or other materials used in connection with that Entry must be limited to code that GE will be entitled to commercially use and exploit, and any open-source-related or third party licensor requirements and restrictions must be disclosed as part of the model submission.

Please do not upload Kaggle-provided data files, but mention which ones are used and how to link them to your agent in the readme.

Regarding non-Kaggle data (Previous leaderboard GRIB files): Zip this data and paste its hash into a text file called 'Data Hash.txt'. Place this text file into your model archive. If your model is in-the-money, we'll contact you to retrieve the non-Kaggle data and validate the hash.

A quick question I meant to ask a while ago. Not that it matters for my own leaderboard position but I think this question is relevant  for other participants also.

My approach is using a semi-stochastic algorithm to discover the best flight paths. This implies that  the submission file is slightly different every time the program is executed with the same parameters and dependent files. Because I won't be able to reproduce the exact same submission file with the submitted model, does this invalidate my solution for the final leardboard submissions?

If you will seed you model's pseudo-random generator with the same value, it will yield the same results.

So, use some predefined seed, and provide it along with your model, in that case your model will be valid for the final submission.

Charango, thanks for the tip, I'll include a predefined seed in my model submission.

I'm not very fond of pseudorandom generators, but I think that different OS or architectures may yield different random sequences.

Anyway, providing a fixed seed may be a good step towards a deterministic result.

I usually close my eyes when I type in a random seed, just to make it more random.

There are some ways stochastic programs can get a little messy. My program periodically writes intermediate results to a file. In the event of a computer glitch, I can pick up close to where I left off without so much time lost. But if I submit a model, I'll need to give some thought to what happens to the state of the pseudo-random number generator in that case, whether writing and reading might cause round-off problems, and whether my process for recovery is in keeping with the rules.

Also, if I let my program grind away till time runs out (assuming that's allowed), I suppose I'll need to record just how many steps it ran.

My code is 2 separate programs (get routes and optimize them). The second programs needs an input of how many waypoints the last flight has. Is this one of the types of parameters which can be changed after code freeze?

Obviously it is possible to re-write the code such that it sees the end of the file without crashing, but it takes time I prefer to invest in getting better results...

I assume it is both valid to upload a text file containing a SHA-256 hash, or a text file containing the standard output of the sha256sum command (SHA-256 hash + file name). Please let us know, if you have a preference on the file format.

The term "bug" is highly subjective. For example, if you crash a plane with the final dataset (due to an unusual starting position, record level headwind or whatever), should that be called a bug and a fix be allowed? I would say no. You could save $$ by cutting corners aggressively, but you should be willing to pay the price for the risk taken. But if *my* planes crashed, no doubt it will start looking like a bug to me :-)

I'm still waiting for an answer to this question. I don't see any answer to either this, or similar questions regarding small differences in outcomes due to random/approximation, so I assume they're accepted.

I got a reply from Charango suggesting to seed the model's pseudo-random generator with the same value. I have implemented this suggestion(in my python code)  and can now generate the same output paths from run to run. So I guess whether your code is deterministic or stochastic, as long as you use the same seed value then it does not matter for the final model submission.