Package it.unimi.dsi.big.util

Class MappedFrontCodedStringBigList

java.lang.Object

java.util.AbstractCollection<K>

it.unimi.dsi.fastutil.objects.AbstractObjectCollection<K>

it.unimi.dsi.fastutil.objects.AbstractObjectBigList<MutableString>

it.unimi.dsi.big.util.MappedFrontCodedStringBigList

All Implemented Interfaces:

BigList<MutableString>, ObjectBigList<MutableString>, ObjectCollection<MutableString>, ObjectIterable<MutableString>, Size64, Stack<MutableString>, Closeable, AutoCloseable, Comparable<BigList<? extends MutableString>>, Iterable<MutableString>, Collection<MutableString>, RandomAccess

public class MappedFrontCodedStringBigList
extends AbstractObjectBigList<MutableString>
implements RandomAccess, Closeable

A memory-mapped version of FrontCodedStringBigList.

This class is functionally identical to FrontCodedStringBigList, but its data is memory-mapped from disk. Only UTF-8 encoding is supported.

To use this class, one first invokes the build(String, int, Iterator) method to generate a property file containing metadata, and two files containing strings and string pointers, respectively. Then, the load(String) method (invoked with the same basename) will return an instance of this class accessing strings and pointers by memory mapping.

Note that for consistency with other classes in this package this class implements a big list of mutable strings; however, for greater flexibility it also implements a getString(long) method and a getArray(long) method.

If you need to build an instance from a (possibly compressed) stream, we suggest to adapt it using FileLinesByteArrayIterable.iterator().

See Also:

• FrontCodedStringBigList

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	MappedFrontCodedStringBigList.PropertyKeys

Nested classes/interfaces inherited from class it.unimi.dsi.fastutil.objects.AbstractObjectBigList

AbstractObjectBigList.ObjectRandomAccessSubList<K extends Object>, AbstractObjectBigList.ObjectSubList<K extends Object>

Field Summary

Fields

Modifier and Type	Field	Description
static final String	BYTE_ARRAY_EXTENSION
protected ByteBigList	byteList	The underlying byte array.
protected final long	n	The number of strings in the list.
protected LongBigList	pointers	The pointers to entire arrays in the list.
static final String	POINTERS_EXTENSION
static final String	PROPERTIES_EXTENSION
protected final int	ratio	The ratio of this front-coded list.
static final long	serialVersionUID

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	MappedFrontCodedStringBigList(long n, int ratio, String byteBigList, String pointers)

Method Summary

Modifier and Type	Method	Description
static void	build(String basename, int ratio, Iterator<byte[]> arrays)	Builds and stores a new memory-mapped front-coded big string list.
void	close()
protected static int	countUTF8Chars(byte[] a)
MutableString	get(long index)	Returns the element at the specified position in this front-coded big list as a mutable string.
void	get(long index, MutableString s)	Returns the element at the specified position in this front-coded big list by storing it in a mutable string.
byte[]	getArray(long index)	Returns the element at the specified position in this front-coded big list as an UTF-8-coded byte array.
String	getString(long index)	Returns the element at the specified position in this front-coded big list as a string.
static MappedFrontCodedStringBigList	load(String basename)	Maps in memory a front-coded string big list starting from a basename.
static void	main(String[] arg)
long	size64()

Methods inherited from class it.unimi.dsi.fastutil.objects.AbstractObjectBigList

add, add, addAll, addAll, addElements, addElements, clear, compareTo, contains, ensureIndex, ensureRestrictedIndex, equals, forEach, getElements, hashCode, indexOf, iterator, lastIndexOf, listIterator, listIterator, peek, pop, push, remove, removeElements, set, setElements, size, size, subList, top, toString

Methods inherited from class java.util.AbstractCollection

containsAll, isEmpty, remove, removeAll, retainAll, toArray, toArray

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.Collection

containsAll, isEmpty, parallelStream, remove, removeAll, removeIf, retainAll, stream, toArray, toArray, toArray

Methods inherited from interface it.unimi.dsi.fastutil.objects.ObjectBigList

addAll, addAll, addAll, addAll, getElements, setElements, setElements, spliterator

Methods inherited from interface it.unimi.dsi.fastutil.Stack

isEmpty

Field Details

serialVersionUID

public static final long serialVersionUID

See Also:

• Constant Field Values

PROPERTIES_EXTENSION

public static final String PROPERTIES_EXTENSION

See Also:

• Constant Field Values

BYTE_ARRAY_EXTENSION

public static final String BYTE_ARRAY_EXTENSION

See Also:

• Constant Field Values

POINTERS_EXTENSION

public static final String POINTERS_EXTENSION

See Also:

• Constant Field Values

n

protected final long n

The number of strings in the list.

ratio

protected final int ratio

The ratio of this front-coded list.

byteList

protected ByteBigList byteList

The underlying byte array.

pointers

protected LongBigList pointers

The pointers to entire arrays in the list.

Constructor Details

MappedFrontCodedStringBigList

protected MappedFrontCodedStringBigList(long n,
 int ratio,
 String byteBigList,
 String pointers)
                                 throws IOException

Throws:

IOException

Method Details

build

public static void build(String basename,
 int ratio,
 Iterator<byte[]> arrays)
                  throws IOException,
ConfigurationException

Builds and stores a new memory-mapped front-coded big string list.

Given a basename, three file with extensions PROPERTIES_EXTENSION, BYTE_ARRAY_EXTENSION and POINTERS_EXTENSION will be generated.

After building a list, you can load it using the same basename.

Parameters:

basename - the basename of the list.

ratio - the ratio.

arrays - an iterator over byte arrays containing UTF-8 encoded-strings.

Throws:

IOException

ConfigurationException

load

public static MappedFrontCodedStringBigList load(String basename)
                                          throws ConfigurationException,
IOException

Maps in memory a front-coded string big list starting from a basename.

Parameters:

basename - the basename of a memory-mapped front-coded string big list.

Returns:

a memory-mapped front-coded string big list.

Throws:

ConfigurationException

IOException

get

public MutableString get(long index)

Returns the element at the specified position in this front-coded big list as a mutable string.

Specified by:

get in interface BigList<MutableString>

Parameters:

index - an index in the list.

Returns:

a MutableString that will contain the string at the specified position. The string may be freely modified.

get

public void get(long index,
 MutableString s)

Returns the element at the specified position in this front-coded big list by storing it in a mutable string.

Parameters:

index - an index in the list.

s - a mutable string that will contain the string at the specified position.

getString

public String getString(long index)

Returns the element at the specified position in this front-coded big list as a string.

Parameters:

index - an index in the list.

Returns:

a String that will contain the string at the specified position.

getArray

public byte[] getArray(long index)

Returns the element at the specified position in this front-coded big list as an UTF-8-coded byte array.

Parameters:

index - an index in the list.

Returns:

a byte array representing in UTF-8 encoding the string at the specified position.

countUTF8Chars

protected static int countUTF8Chars(byte[] a)

size64

public long size64()

Specified by:

size64 in interface Size64

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException

main

public static void main(String[] arg)
                 throws IOException,
JSAPException,
ConfigurationException,
ClassNotFoundException,
IllegalArgumentException,
SecurityException

Throws:

IOException

JSAPException

ConfigurationException

ClassNotFoundException

IllegalArgumentException

SecurityException