Re: py-lmdb often gets segmentation faults

On Sat, 28 Jan 2023 21:11:06 -0000 (UTC)
Stuart Henderson <stu.lists@spacehopper.org> wrote:

> Thanks for the detailed report, thinking about what information
> might be useful, and including it directly in your email.
>
> On 2023-01-28, Roger Marsh <r.m@rmswch.plus.com> wrote:
> > Package lmdb-0.9.29 was installed; and py-lmdb, as a --user install, from PyPi so lmdb can be used from Python3.9.
>
> py-lmdb bundles its own copy of LMDB and uses it by default, or you
> can set extra flags to use the system version.
>
> Normally the bundled one is ok, but the LMDB library assumes an OS with
> coherent file and mmap access (often referred to as 'unified buffer
> cache') which OpenBSD lacks. The LMDB port (and OpenLDAP) are patched to
> force MDB_WRITEMAP so all io goes through mmap. This has some drawbacks
> (and some advantages in certain circumstances) but as far as we know
> it's the only way we can run it. Obviously the copy bundled in py-lmdb
> doesn't have this patch.
>
> "Use a writeable memory map unless MDB_RDONLY is set. This is
> faster and uses fewer mallocs, but loses protection from application
> bugs like wild pointer writes and other bad updates into the database.
> Incompatible with nested transactions. Do not mix processes with and
> without MDB_WRITEMAP on the same environment. This can defeat durability
> (mdb_env_sync etc)."
>
> (If I understand correctly, LMDB normally writes through file access,
> reads through mmap).
>
> > Behaviour seemed erratic, including segmentation faults, but for a
> > while it seemed possible to avoid these by using alternative methods
> > supported by py-lmdb.
>
> Yes this sounds about the expected behaviour for the UBC problem.
>
> You can try the port I've just posted to ports@ which uses the system
> lmdb library. https://marc.info/?l=openbsd-ports&m=167493896320807&w=2
> Alternatively set the flags from MAKE_ENV in that port while building
> py-lmdb manually.
>
>

Thanks for the explanation and the port.