Bug #2955
closedlua issues on arm (fedora:29)
Description
The suricata-verify 'lua-output-dns' test fails because the produced logfile contains some strange values:
05/24/2016-23:27:01.960780 [**] Query TX 2b2ea9628 [**] client-cf.dropbox.com [**] A [**] 10.16.1.11:53679 -> 10.16.1.1:53 05/24/2016-23:27:02.832606 [**] Query TX 2b10a9628 [**] block.dropbox.com [**] A [**] 10.16.1.11:49697 -> 10.16.1.1:53 05/24/2016-23:27:04.653864 [**] Query TX 2b06a9628 [**] client-cf.dropbox.com [**] A [**] 10.16.1.11:57634 -> 10.16.1.1:53 10/14/2016-15:40:21.889830 [**] Query TX 2b42aa6a8 [**] d98cf633-97be-406f-9e39-bd8fc0cbdea4.com [**] A [**] 10.16.1.11:40697 -> 10.16.1.1:53 05/24/2016-23:27:02.333141 [**] Query TX 2b2ea9628 [**] client-cf.dropbox.com [**] A [**] 10.16.1.11:53679 -> 10.16.1.1:53 05/24/2016-23:27:02.333141 [**] Response TX 2b2ea9628 [**] client-cf.dropbox.com [**] A [**] TTL 77968877786497092 [**] 52.85.112.21 [**] 10.16.1.11:53679 -> 10.16.1.1:53 05/24/2016-23:27:03.085375 [**] Query TX 2b10a9628 [**] codemonkey.net [**] A [**] 10.16.1.11:33458 -> 10.16.1.1:53 05/24/2016-23:27:04.654238 [**] Query TX 2b06a9628 [**] client-cf.dropbox.com [**] A [**] 10.16.1.11:57634 -> 10.16.1.1:53 05/24/2016-23:27:04.654238 [**] Response TX 2b06a9628 [**] client-cf.dropbox.com [**] A [**] TTL 77968877786497092 [**] 52.85.112.21 [**] 10.16.1.11:57634 -> 10.16.1.1:53 10/14/2016-15:40:21.971664 [**] Query TX 2b42aa6a8 [**] d98cf633-97be-406f-9e39-bd8fc0cbdea4.com [**] A [**] 10.16.1.11:40697 -> 10.16.1.1:53 10/14/2016-15:40:21.971664 [**] Response TX 2b42aa6a8 [**] NXDOMAIN [**] 10.16.1.11:40697 -> 10.16.1.1:53 10/14/2016-15:40:21.971664 [**] Response TX 2b42aa6a8 [**] com [**] SOA [**] TTL 77968877786497092 [**] 10.16.1.11:40697 -> 10.16.1.1:53 05/24/2016-23:27:03.213624 [**] Query TX 2b10a9628 [**] block.dropbox.com [**] A [**] 10.16.1.11:49697 -> 10.16.1.1:53 05/24/2016-23:27:03.213624 [**] Response TX 2b10a9628 [**] block.g1.dropbox.com [**] A [**] TTL 77968877786497092 [**] 45.58.70.33 [**] 10.16.1.11:49697 -> 10.16.1.1:53 05/24/2016-23:27:03.213624 [**] Response TX 2b10a9628 [**] block.dropbox.com [**] CNAME [**] TTL 77968877786497092 [**] block.g1.dropbox.com [**] 10.16.1.11:49697 -> 10.16.1.1:53 05/24/2016-23:27:03.493333 [**] Query TX 2b10a9d48 [**] codemonkey.net [**] A [**] 10.16.1.11:33458 -> 10.16.1.1:53 05/24/2016-23:27:03.493333 [**] Response TX 2b10a9d48 [**] codemonkey.net [**] A [**] TTL 77968877786497092 [**] 104.131.202.103 [**] 10.16.1.11:33458 -> 10.16.1.1:53
The id's are wrong and the ttl values look rather suspect.
Setup:
Docker on ARM (32 bit) with fedora:29 image.
Test 'dns-lua-rules' also fails. The EVE log DNS records look normal, so I wonder if the lua-rust layer is mangling types.
Updated by Victor Julien over 5 years ago
- Status changed from New to Assigned
- Assignee set to Jason Ish
- Priority changed from Normal to High
- Target version set to 5.0rc1
Jason can you have a look? I can see that Rust has the values correctly, but somehow they get mangled when the Lua script accesses them. Not really sure how to analyse.
Updated by Victor Julien about 5 years ago
- Target version changed from 5.0rc1 to 5.0.0
Updated by Jason Ish about 5 years ago
After some testing I can confirm that this test fails on "ARMv7 Processor rev 2", with an Ubuntu 18.04 as host OS and Fedora 29 running inside Docker.
The test does pass when running on the host Ubuntu 18.04, as well as Ubuntu 18.04 running inside Docker, so the problem appears to be related to Fedora. I doubt that running in or out of Docker would make a difference.
Updated by Victor Julien about 5 years ago
- Priority changed from High to Normal
- Target version changed from 5.0.0 to TBD
Ok, thanks Jason. I suggest we ignore the issue for now. If a fedora:30 image ever comes out, or a fedora:31 image, we can retest.
Updated by Jason Ish about 5 years ago
The default lua package in Fedora 29 is Lua 5.3.5. Fedora also ships a "compat-lua-libs" and "compat-lua-devel" which is Lua 5.1.5. When these packages are installed, and "lua-devel" is not installed, then the tests pass.
It is something in the difference betwen Lua 5.1.5 and 5.3.5 on Fedora 29, on ARM32 at least. I'd probably have to build Lua myself to find out more (compare compile time options, etc).
Updated by Victor Julien about 5 years ago
Okay that is interesting. I think it would be good to investigate lua 5.3+ in general. We picked 5.1 initially (maybe it was the most recent, or perhaps because of luajit), but never looked into how never versions are different/better, etc.
Updated by Jason Ish about 5 years ago
So the root of the problem is the change of integer size from 32 bits to 64 bits between Lua 5.1 and 5.3. This issue can also be seen on i386, but i386 was a little more forgiving of using 64 bit at the FFI layer, probably due to some architecture dependent stuff inside Lua.
It looks like the options are:
- Require to Lua 5.1. Of note it looks like CentOS 8 and RHEL 8 don't have a Lua 5.1 compat package. That may change as CentOS 8 is going to see wider use soon.
- Do conditional code based on the integer size - we can use the Lua version number to get a good idea. This will cover this case, but there might be other incompatibilities that might bite us.
- Vendor the lua code, as you would see in a commercial application (ie: a game) that provides Lua scripting support.
Perhaps there are other options as well that I have not considered.
It is important to note that the Lua developers, from what I can tell embrace the ability to make breaking changes between versions such as 5.1, 5.2, 5.3, etc. Which does make it hard for us, or any user who does not vendor the Lua code to provide guarantees on what version is available.
Updated by Victor Julien about 5 years ago
- Status changed from Assigned to Closed
- Target version changed from TBD to 5.0.0
- Label Needs backport added
Updated by Victor Julien almost 5 years ago
- Copied to Bug #3325: lua issues on arm (fedora:29) (4.1.x) added