Comments on: The Problem with /Direct Port Access
REBOL Technologies

Comments on: The Problem with /Direct Port Access

Carl Sassenrath, CTO
REBOL Technologies
26-Aug-2005 23:33 GMT

Article #0198
Main page || Index || Prior Article [0197] || Next Article [0199] || 1 Comments || Send feedback

There has been a long standing problem with /direct access I/O ports when used for random access (seeking) within files. This has made access to large files difficult.

Update:

This problem has been fixed by adding a new /seek mode to REBOL. Read the next article for details.

Let me illustrate the problem with an example:

write %example.txt "1234567890"

port: open/direct/binary %example.txt
print as-string copy/part port 2
12
print as-string copy/part port 2
34
port: head port
print as-string copy/part port 2
56

As you can see, the head function did not have any effect on the result. It was ignored. You can also see that the copy function is automatically advancing the read position within the file.

This behavior is different from ports opened without /direct access (and it is also different from the standard REBOL concept of a series). This example shows why:

port: open/binary %example.txt
print as-string copy/part port 2
12
print as-string copy/part port 2
12
port: skip port 2
print as-string copy/part port 2
34
port: head port
print as-string copy/part port 2
12

Here without the /direct mode, the port acts just like a standard REBOL series (as you would expect).

What makes the problem worse is that the /direct mode does not perform an actual seek on the file. If you open a large file (say 100 MB) and skip 50 MB forward, the direct port will actually perform a read of all 50 MB of data to reach the desired position. That slows down random access within large files. (And skip may also hang at times.)

In my opinion, the implementation of /direct mode on files is broken. The question is how should it be fixed. There are two choices:

  1. Fix /direct mode to work the same way as non-direct mode.
  2. Add a new mode called /seek.

Choice 1 seems like the perfect solution, but there may be quite a few applications that open files in direct mode, and they will break if they expect the file position to move forward after each copy function.

So, it looks like we will need to add a /seek mode to ports. We did this in the new asynchronous version of REBOL, and it worked well. But the current releases use a different implementation of ports, so we will need to design a new solution.

We've raised this issue to a high priority bug fix, because REBOL absolutely needs to have a better way to randomly access large files. You can look for a new beta test release of REBOL very soon.

1 Comments

Comments:

Buzzard
26-Dec-2013 10:21:42
Will this be a beta of rebol 2 or rebol 3?

Post a Comment:

You can post a comment here. Keep it on-topic.

Name:

Blog id:

CS-0198


Comment:


 Note: HTML tags allowed for: b i u li ol ul font p br pre tt blockquote
 
 

This is a technical blog related to the above topic. We reserve the right to remove comments that are off-topic, irrelevant links, advertisements, spams, personal attacks, politics, religion, etc.

Updated 25-Nov-2017   -   Copyright Carl Sassenrath   -   WWW.REBOL.COM   -   Edit   -   Blogger Source Code