Executors are classes that deterministically transform programs in domain specific languages into denotations. We have one executor defined for each language-domain pair that we handle.
This class evaluates lambda-DCS logical forms by calling out to SEMPRE, where the particular lambda-DCS language we use was defined. It’s a huge pain to have to rely on a call to a java subprocess, but it’s way easier than trying to write our own executor for the language.
Because of how the SEMPRE executor works, we need to have access to the original table on disk, and we need to pass in the full lisp “example” string that’s given in the dataset. SEMPRE parses the example string (which includes a path to the table), reads the table, executes the logical form in the context of the table, and compares the answer produced to the answer specified in the example string.
We don’t even get back the denotation of the logical form here, because then we’d have to do a comparison with the correct answer, and that’s a bit messy - better to just let SEMPRE do it. This is why we only provide a
evaluate_logical_form()method that returns a
boolinstead of an
executemethod returning an answer. You might think that if we got the answer back, we could at least use this for a demo. The sad thing is that even that doesn’t work, because this executor relies on having the table for the example already accessible on disk, which we don’t have in the case of a demo - we have to do extra stuff there to get it to work, including writing the table to disk so that SEMPRE can access it! It’s all a bit of a mess.
evaluate_logical_form(logical_form: str, example_lisp_string: str) → bool¶
This class evaluates SQL queries by connecting to a SQLite database. Because SQLite is disk-based we just need to provide one file with the location. We execute the predicted SQL query and the labeled queries against the database and check if they execute to the same table.
evaluate_sql_query(predicted_sql_query: str, sql_query_labels: List[str]) → int¶