The Problem with /Direct Port Access

There has been a long standing problem with /direct access I/O ports when used for random access (seeking) within files. This has made access to large files difficult.

Let me illustrate the problem with an example:

write %example.txt "1234567890"

port: open/direct/binary %example.txt
print as-string copy/part port 2
12
print as-string copy/part port 2
34
port: head port
print as-string copy/part port 2
56

As you can see, the head function did not have any effect on the result. It was ignored. You can also see that the copy function is automatically advancing the read position within the file.

This behavior is different from ports opened without /direct access (and it is also different from the standard REBOL concept of a series). This example shows why:

port: open/binary %example.txt
print as-string copy/part port 2
12
print as-string copy/part port 2
12
port: skip port 2
print as-string copy/part port 2
34
port: head port
print as-string copy/part port 2
12

Here without the /direct mode, the port acts just like a standard REBOL series (as you would expect).

What makes the problem worse is that the /direct mode does not perform an actual seek on the file. If you open a large file (say 100 MB) and skip 50 MB forward, the direct port will actually perform a read of all 50 MB of data to reach the desired position. That slows down random access within large files. (And skip may also hang at times.)

In my opinion, the implementation of /direct mode on files is broken. The question is how should it be fixed. There are two choices:

1. Fix /direct mode to work the same way as non-direct mode.
2. Add a new mode called /seek.

Choice 1 seems like the perfect solution, but there may be quite a few applications that open files in direct mode, and they will break if they expect the file position to move forward after each copy function.

So, it looks like we will need to add a /seek mode to ports. We did this in the new asynchronous version of REBOL, and it worked well. But the current releases use a different implementation of ports, so we will need to design a new solution.

We've raised this issue to a high priority bug fix, because REBOL absolutely needs to have a better way to randomly access large files. You can look for a new beta test release of REBOL very soon.

1 Comments

Updated 9-Mar-2024   -   Copyright Carl Sassenrath   -   WWW.REBOL.COM   -   Edit   -   Blogger Source Code