Can't specify range for p_timestamp

[nalcabio-tom]

For example, this query "works" but only returns data from the past ten minutes (the default in parseable when no time range is specified).

SELECT "ctx#timestamp" AS "ctx#timestamp", "ctx#event" AS "ctx#event", "data#device-state" AS "data#device-state", p_timestamp AS p_timestamp
FROM nalca1
WHERE "ctx#event" = 'bio.nalca.components.hw/set-device-state' AND p_timestamp >= '2025-01-01 00:00:00.000000' AND p_timestamp < '2025-06-06 00:00:00.000000'
LIMIT 100

How do I specify a range for p_timestamp? See also #5 and #6

[AdheipSingh]

@nalcabio-tom thanks for reporting this.

Can you test with 0.1.6 version. I pushed some changes.

Thanks

[nalcabio-tom]

@AdheipSingh Sorry, this is still not working as expected. Still the data is being restricted to only the past ten minutes as I tried to explain at #5 (comment).

Here's the data that's in parseable:

However this data does not show up in superset:

Only data that was added to parseable within the past ten minutes shows up in superset.

This is probably an unrelated issue:

=== QUERY ERROR ===
400 Client Error: Bad Request for url: https://thing1.flamingo-daggertooth.ts.net:9005/api/v1/query
================

2025-03-07 14:53:02,022:WARNING:superset.views.error_handling:SupersetErrorsException
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.10/site-packages/flask_appbuilder/security/decorators.py", line 95, in wraps
    return f(self, *args, **kwargs)
  File "/app/superset/views/base_api.py", line 119, in wraps
    duration, response = time_function(f, self, *args, **kwargs)
  File "/app/superset/utils/core.py", line 1364, in time_function
    response = func(*args, **kwargs)
  File "/app/superset/views/base_api.py", line 91, in wraps
    return f(self, *args, **kwargs)
  File "/app/superset/utils/log.py", line 303, in wrapper
    value = f(*args, **kwargs)
  File "/app/superset/sqllab/api.py", line 409, in execute_sql_query
    command_result: CommandResult = command.run()
  File "/app/superset/utils/decorators.py", line 266, in wrapped
    return on_error(ex)
  File "/app/superset/utils/decorators.py", line 236, in on_error
    raise ex
  File "/app/superset/utils/decorators.py", line 259, in wrapped
    result = func(*args, **kwargs)
  File "/app/superset/commands/sql_lab/execute.py", line 105, in run
    status = self._run_sql_json_exec_from_scratch()
  File "/app/superset/commands/sql_lab/execute.py", line 158, in _run_sql_json_exec_from_scratch
    return self._sql_json_executor.execute(
  File "/app/superset/sqllab/sql_json_executer.py", line 111, in execute
    raise SupersetErrorsException(
superset.exceptions.SupersetErrorsException: [SupersetError(message='base error: Query execution failed: 400 Client Error: Bad Request for url: https://thing1.flamingo-daggertooth.ts.net:9005/api/v1/query', error_type=<SupersetErrorType.GENERIC_DB_ENGINE_ERROR: 'GENERIC_DB_ENGINE_ERROR'>, level=<ErrorLevel.ERROR: 'error'>, extra={'engine_name': None, 'issue_codes': [{'code': 1002, 'message': 'Issue 1002 - The database returned an unexpected error.'}]})]

[AdheipSingh]

On demo.parseable.com we have ingress-nginx dataset which ingests nginx logs. I am testing this query to make sure it's working. You can run this same query on superset.parseable.com to check the results.

SELECT status AS status, method AS method, remote_addr AS remote_addr, p_timestamp AS p_timestamp FROM "ingress-nginx" 
WHERE p_timestamp >= '2025-02-15 00:00:00.000000' AND p_timestamp < '2025-03-05 00:00:00.000000' 
limit 10

In your logs you should see something like original query and modified query can you give a dump of those logs.

[nalcabio-tom]

@AdheipSingh I really appreciate your continued attention to this issue, but I still don't think we both understand the problem in the same way. Could you please check that in your example above you get back any results that are older than 10 minutes? The problem is not with the sql query. The problem is with the json payload that is posted to parseable. Separate from the sql query, the start and end times must be specified as well. From my comment at #5 (comment)

It seems to me that these start and end times are never sent to parseable.

[nalcabio-tom]

Please note the Time Range below for running the query above:

[nalcabio-tom]

Someone recommended using jinja2 templates, https://superset.apache.org/docs/configuration/sql-templating Does this make sense as a solution to you @AdheipSingh?

[nalcabio-tom]

@AdheipSingh , I'd be happy to help troubleshoot this, but we've decided to switch to Elasticsearch and Kibana. Feel free to close this issue if you'd like. Thanks!