dataloader

Index

    1. READMEmd[source]
  2. src
    1. 0packagelisp[source]
    2. 1genericlisp[source]
    3. 2loadlisp[source]
    4. 2savelisp[source]
    5. 3apilisp[source]

READMEmd[source]

Dataloader - A universal loader library for various data formats for images/audio img

This is what you might have needed for a long time. It loads a file in an arbitrary data format into a numcl-compatible array. File types are automatically detected by the mime type information obtained by libmagic.

Supported files are currently png, jpg, tiff, csv, tsv, wav, npy. We plan to include support for bmp, gif, npz. This library relies on existing libraries to load the files, such as cl-png, retrospectiff, cl-csv, but provides a consistent and natural array-based interface.

An image file is loaded into a NUMCL-compatible array with corresponding width, height and channels.

A sound file is loaded into a 1D array (monoral), or a 2D array with 2 channels (stereo). It provides a wrapper over cl-wav and correctly returns a (unsigned-byte 16) arrays for a 16-bit PCM file, which is not performed by cl-wav. The save method also automatically generates RIFF headers, which is not performed by cl-wav.

A numpy file is loaded with the corresponding width and height using numpy-file-format library.

Load

DATALOADER:LOAD
  [symbol]

LOAD names a compiled function:
  Lambda-list: (FILE &REST ARGS &KEY
                (MIME (MAGICFFI:PATHNAME-MIME-TYPE FILE))
                
                ;; jpeg options
                (DECODE-FRAME T) CACHED-SOURCE-P (COLORSPACE-CONVERSION T) BUFFER
                
                ;; csv options
                ESCAPE-MODE NEWLINE
                TRIM-OUTER-WHITESPACE QUOTED-EMPTY-STRING-IS-NIL
                UNQUOTED-EMPTY-STRING-IS-NIL
                (ESCAPE CL-CSV:*QUOTE-ESCAPE*) 'CL-CSV:*QUOTE*
                (SEPARATOR CL-CSV:*SEPARATOR*) SKIP-FIRST-P SAMPLE
                DATA-MAP-FN MAP-FN ROW-FN CSV-READER TYPE)

This function loads a file and returns a numcl-compatible array. The file format can be specified manually in the mime string, but by default detected by libmagic.

Save

DATALOADER:SAVE
  [symbol]

SAVE names a compiled function:
  Lambda-list: (ARRAY FILE &REST ARGS &KEY (MIME (PATHNAME-TYPE FILE))
  
                ;; wav options
                (BITS-PER-SECOND 44100))

This function saves the array into a file, using the file name extension or the additional mime argument.

Installation

Available from quicklisp.

Dependencies

This library is at least tested on implementation listed below:

  • SBCL 1.4.12 on X86-64 Linux 4.4.0-141-generic (author’s environment)

Also, it depends on the following libraries:

  • cl-wav :

  • cl-png :

  • cl-jpeg :

  • retrospectiff :

  • iterate by ** : Jonathan Amsterdam’s iterator/gatherer/accumulator facility

  • alexandria by Nikodemus Siivola , and others. : Alexandria is a collection of portable public domain utilities.

  • trivia by Masataro Asai : NON-optimized pattern matcher compatible with OPTIMA, with extensible optimizer interface and clean codebase

Author

  • Masataro Asai (guicho2.71828@gmail.com)

Copyright (c) 2019 Masataro Asai (guicho2.71828@gmail.com)

License

Licensed under the LLGPL License.

src

0packagelisp[source]

define-package
dataloader.mime

A package containing symbols that are supported by dataloader. Each symbol has a name read from a mime type string, e.g., "image/png" -> IMAGE/PNG . The symbols are used as an eql specializer for a generic function dataloader:perform-load and dataloader:perform-save.

define-package
dataloader

The main package for dataloader.

1genericlisp[source]

defgeneric
perform-load
file mime &rest args &key &allow-other-keys

A generic function that is defined for each mime type symbol in DATALOADER.MIME package.

defmethod
perform-load
(file string) mime &rest args &key &allow-other-keys

Convert a namestring into a pathanme

defmethod
perform-load
file (mime string) &rest args &key &allow-other-keys

Convert a mime string into a symbol

defmacro
define-load-method
(filevar mime-string-or-strings &rest lambda-list) &body body

A wrapper over defmethod. Made primarily because defmethod does not support (or (eql ...)) or member as a specializer. mime-string-or-strings is a string or a list of strings, where each string specifies a mime type e.g. "image/png". lambda-list contains a keyword argument portion of the congruent lambda list of defgeneric. In order to have a compatible lambda-list with the generic function, it adds &rest argument if not present, and &allow-other-keys when &key is present and &allow-other-keys is missing.

defgeneric
perform-save
array file mime &rest args &key &allow-other-keys

A generic function that is defined for each mime type symbol in DATALOADER.MIME package.

defmethod
perform-save
array (file string) mime &rest args &key &allow-other-keys

Convert a namestring into a pathanme

defmethod
perform-save
array file (mime string) &rest args &key &allow-other-keys

Convert a mime string into a symbol

defmacro
define-save-method
(arrayvar filevar mime-string-or-strings &rest lambda-list) &body body

A wrapper over defmethod. Made primarily because defmethod does not support (or (eql ...)) or member as a specializer. mime-string-or-strings is a string or a list of strings, where each string specifies a mime type e.g. "image/png". lambda-list contains a keyword argument portion of the congruent lambda list of defgeneric. In order to have a compatible lambda-list with the generic function, it adds &rest argument if not present, and &allow-other-keys when &key is present and &allow-other-keys is missing.

2loadlisp[source]

defmethod
perform-load
file (mime (eql 'png)) &rest args0

defmethod
perform-load
file (mime (eql 'image/png)) &rest args0

defmethod
perform-load
file (mime (eql 'jpg)) &rest args &key buffer (colorspace-conversion t) cached-source-p (decode-frame t) &allow-other-keys

defmethod
perform-load
file (mime (eql 'jpeg)) &rest args &key buffer (colorspace-conversion t) cached-source-p (decode-frame t) &allow-other-keys

defmethod
perform-load
file (mime (eql 'image/jpeg)) &rest args &key buffer (colorspace-conversion t) cached-source-p (decode-frame t) &allow-other-keys

defmethod
perform-load
file (mime (eql 'tiff)) &rest args0

defmethod
perform-load
file (mime (eql 'image/tiff)) &rest args0

defmethod
perform-load
file (mime (eql 'csv)) &rest args &key type csv-reader row-fn map-fn data-map-fn sample skip-first-p (separator *separator*) '*quote* (escape *quote-escape*) unquoted-empty-string-is-nil quoted-empty-string-is-nil trim-outer-whitespace newline escape-mode &allow-other-keys

defmethod
perform-load
file (mime (eql 'tsv)) &rest args &key type csv-reader row-fn map-fn data-map-fn sample skip-first-p (separator *separator*) '*quote* (escape *quote-escape*) unquoted-empty-string-is-nil quoted-empty-string-is-nil trim-outer-whitespace newline escape-mode &allow-other-keys

defmethod
perform-load
file (mime (eql 'application/csv)) &rest args &key type csv-reader row-fn map-fn data-map-fn sample skip-first-p (separator *separator*) '*quote* (escape *quote-escape*) unquoted-empty-string-is-nil quoted-empty-string-is-nil trim-outer-whitespace newline escape-mode &allow-other-keys

defmethod
perform-load
file (mime (eql 'application/x-csv)) &rest args &key type csv-reader row-fn map-fn data-map-fn sample skip-first-p (separator *separator*) '*quote* (escape *quote-escape*) unquoted-empty-string-is-nil quoted-empty-string-is-nil trim-outer-whitespace newline escape-mode &allow-other-keys

defmethod
perform-load
file (mime (eql 'application/vnd.ms-excel)) &rest args &key type csv-reader row-fn map-fn data-map-fn sample skip-first-p (separator *separator*) '*quote* (escape *quote-escape*) unquoted-empty-string-is-nil quoted-empty-string-is-nil trim-outer-whitespace newline escape-mode &allow-other-keys

defmethod
perform-load
file (mime (eql 'text/csv)) &rest args &key type csv-reader row-fn map-fn data-map-fn sample skip-first-p (separator *separator*) '*quote* (escape *quote-escape*) unquoted-empty-string-is-nil quoted-empty-string-is-nil trim-outer-whitespace newline escape-mode &allow-other-keys

defmethod
perform-load
file (mime (eql 'text/comma-separated-values)) &rest args &key type csv-reader row-fn map-fn data-map-fn sample skip-first-p (separator *separator*) '*quote* (escape *quote-escape*) unquoted-empty-string-is-nil quoted-empty-string-is-nil trim-outer-whitespace newline escape-mode &allow-other-keys

defmethod
perform-load
file (mime (eql 'text/x-csv)) &rest args &key type csv-reader row-fn map-fn data-map-fn sample skip-first-p (separator *separator*) '*quote* (escape *quote-escape*) unquoted-empty-string-is-nil quoted-empty-string-is-nil trim-outer-whitespace newline escape-mode &allow-other-keys

defmethod
perform-load
file (mime (eql 'text/x-comma-separated-values)) &rest args &key type csv-reader row-fn map-fn data-map-fn sample skip-first-p (separator *separator*) '*quote* (escape *quote-escape*) unquoted-empty-string-is-nil quoted-empty-string-is-nil trim-outer-whitespace newline escape-mode &allow-other-keys

defmethod
perform-load
file (mime (eql 'text/tab-separated-values)) &rest args &key type csv-reader row-fn map-fn data-map-fn sample skip-first-p (separator *separator*) '*quote* (escape *quote-escape*) unquoted-empty-string-is-nil quoted-empty-string-is-nil trim-outer-whitespace newline escape-mode &allow-other-keys

defmethod
perform-load
file (mime (eql 'text/plain)) &rest args &key type csv-reader row-fn map-fn data-map-fn sample skip-first-p (separator *separator*) '*quote* (escape *quote-escape*) unquoted-empty-string-is-nil quoted-empty-string-is-nil trim-outer-whitespace newline escape-mode &allow-other-keys

defmethod
perform-load
file (mime (eql 'wav)) &rest args0

defmethod
perform-load
file (mime (eql 'audio/wav)) &rest args0

defmethod
perform-load
file (mime (eql 'audio/x-wav)) &rest args0

defmethod
perform-load
file (mime (eql 'audio/wave)) &rest args0

defmethod
perform-load
file (mime (eql 'audio/x-wave)) &rest args0

defmethod
perform-load
file (mime (eql 'audio/vnd.wave)) &rest args0

(all documentation missing)

defmethod
perform-load
file (mime (eql 'npy)) &rest args0

libmagic can detect numpy file, but does not return a meaningful mime type

defmethod
perform-load
file (mime (eql 'application/octet-stream)) &rest args

fallback method for application/octet-stream mime type. Since not much can be inferred from this mime type, it calls itself with pathname extension as the second argument.

2savelisp[source]

defmethod
perform-save
data file (mime (eql 'png)) &rest args0

defmethod
perform-save
data file (mime (eql 'image/png)) &rest args0

defmethod
perform-save
data file (mime (eql 'jpg)) &rest args0

defmethod
perform-save
data file (mime (eql 'jpeg)) &rest args0

defmethod
perform-save
data file (mime (eql 'image/jpeg)) &rest args0

defmethod
perform-save
data file (mime (eql 'tiff)) &rest args0

defmethod
perform-save
data file (mime (eql 'image/tiff)) &rest args0

defmethod
perform-save
data file (mime (eql 'csv)) &rest args0

defmethod
perform-save
data file (mime (eql 'application/csv)) &rest args0

defmethod
perform-save
data file (mime (eql 'application/x-csv)) &rest args0

defmethod
perform-save
data file (mime (eql 'text/csv)) &rest args0

defmethod
perform-save
data file (mime (eql 'text/comma-separated-values)) &rest args0

defmethod
perform-save
data file (mime (eql 'text/x-csv)) &rest args0

defmethod
perform-save
data file (mime (eql 'text/x-comma-separated-values)) &rest args0

defmethod
perform-save
data file (mime (eql 'tsv)) &rest args0

defmethod
perform-save
data file (mime (eql 'text/tab-separated-values)) &rest args0

defmethod
perform-save
data file (mime (eql 'wav)) &rest args0 &key (bits-per-second 44100) &allow-other-keys

defmethod
perform-save
data file (mime (eql 'audio/wav)) &rest args0 &key (bits-per-second 44100) &allow-other-keys

defmethod
perform-save
data file (mime (eql 'audio/x-wav)) &rest args0 &key (bits-per-second 44100) &allow-other-keys

defmethod
perform-save
data file (mime (eql 'audio/wave)) &rest args0 &key (bits-per-second 44100) &allow-other-keys

defmethod
perform-save
data file (mime (eql 'audio/x-wave)) &rest args0 &key (bits-per-second 44100) &allow-other-keys

defmethod
perform-save
data file (mime (eql 'audio/vnd.wave)) &rest args0 &key (bits-per-second 44100) &allow-other-keys

defmethod
perform-save
data file (mime (eql 'npy)) &rest args0

(all documentation missing)

3apilisp[source]

defun
load
file &rest args &key (mime (pathname-mime-type file)) (decode-frame t) cached-source-p (colorspace-conversion t) buffer escape-mode newline trim-outer-whitespace quoted-empty-string-is-nil unquoted-empty-string-is-nil (escape *quote-escape*) '*quote* (separator *separator*) skip-first-p sample data-map-fn map-fn row-fn csv-reader type

Loads a file and returns a numcl-compatible array. The file format is detected by libmagic, but it can optionally be specified.

defun
save
array file &rest args &key (mime (pathname-type file)) (bits-per-second 44100)

This function saves the array into a file, using the format specified by the file name extension or by the mime type.