GlueGen Struct Generator Update *API Change* .. etc

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

GlueGen Struct Generator Update *API Change* .. etc

Sven Gothel
Administrator
I made a detailed overhaul of GlueGen's Struct Code Generator.

(Next desired step: Adding callback-generator C -> Java for C-function callbacks)

Documentation:
- New document covering GlueGen in general
  <https://jogamp.org/cgit/gluegen.git/about/doc/GlueGen_Mapping.md>

- Specifically Struct Mapping (what has changes)
  <https://jogamp.org/cgit/gluegen.git/about/doc/GlueGen_Mapping.md#struct-mapping>

- Actual JavaEmitter commit with details
  <https://jogamp.org/cgit/gluegen.git/commit/?id=8b127c4c1dd26fcb1756805ddb83729203161f78>

- All git changes visible as usual here
  <https://jogamp.org/cgit/gluegen.git/log/>

The JOAL, JOGL and JOCL changes were minimal:
<https://jogamp.org/cgit/joal.git/log/>
<https://jogamp.org/cgit/jogl.git/log/>
<https://jogamp.org/cgit/jocl.git/log/>

Further I updated
- GlueGen git about <https://jogamp.org/cgit/gluegen.git/about/>
- GlueGen web <https://jogamp.org/gluegen/www/>

Please discuss ... below the copy & paste of the
main git commit message of JavaEmitter commit with details
<https://jogamp.org/cgit/gluegen.git/commit/?id=8b127c4c1dd26fcb1756805ddb83729203161f78>

GlueGen Revised Struct Mapping (esp pointer to array or single element), Struct String Charset, .. and Documentation

- Documentation:
    - Added README.md
      Let's have a proper face for the git repo

    - Added doc/GlueGen_Mapping.md (and its html conversion doc/GlueGen_Mapping.html)
      Created a new document covering application and implementation details suitable for users/devs.

    - Added doc/JogAmpMacOSVersions.md conversion to doc/JogAmpMacOSVersions.html

    - Updated www/index.html

- Use *CodeUnit instead of PrintWriter, representing a Java or C code unit covering a set of functions and structs.
    The CCodeUnit also handles common code shared by its unit across functions etc.

- Dropping 'static initializer', as its no more required
    due to simplified `JVMUtil_NewDirectByteBufferCopy()` variant.

- Revised Struct Mapping:
  - Pure Java implementation to map primitive and struct fields within a struct
    by utilizing ElementBuffer.

    Only 'Function Pointer' fields within a struct require native code.

    Exposes `static boolean usesNativeCode()` to query whether native code is used/required.

  - Transparent native memory address API

    Expose `long getDirectBufferAddress()` and `static TK_Struct derefPointer(long addr)`,
    allowing to
    - pass the native struct-pointer with native code
    - reconstruct the struct from a native struct-pointer
    - have a fully functional `TK_Struct.derefPointer(struct.getDirectBufferAddress())` cycle.

  - Add 'boolean is<Val>Null() to query whether a pointer (array) is NULL

  - *Changed* array get/set method for more flexibility alike `System.arraycopy(src, srcPos, dest, destPos, len)`,
    where 'src' is being dropped for the getter and 'dest' is being dropped for the setter
    as both objects are reflected by the struct instance.

  - *Changed* `get<Val>ArrayLength()` -> `get<Val>ElemCount()` for clarity

  - Considering all ConstElemCount values with config 'ReturnedArrayLength <int>'
    to be owned by native code -> NativeOwnership -> Not changing the underlying memory region!
    JavaOwnership is considered for all pointer-arrays not of NativeOwnership.
    Hence any setter on a NativeOwnership pointer-array will fail with non-matching elem-count.

  - Add 'release<Val>()' for JavaOwnership pointer-arrays,
    allowing to release the Java owned native memory incl. null-ing pointer and set<Val>ElemCount(0).

  - Support setter for 'const <type>*' w/ JavaOwnership, i.e. pointer to const value of a primitive or struct,
    setter and getter using pointer to array or single element in general.

  - Added Config `ImmutableAccess symbol` to disable all setter for whole struct or a field

  - Added Config `MaxOneElement symbol` to restrict a pointer to maximum one element and unset
    initial value (zero elements)

  - Added Config `ReturnsStringOnly symbol` to restrict mapping only to a Java String,
    dropping the ByteBuffer variant for 'char'

  - String mapping default is UTF-8 and can be read and set via [get|set]Charset(..) per class.

  - Dynamic string length retrieval in case no `ReturnedArrayLength` has been configured
    has changed from `strlen()` to `strnlen(aptr, max_len)` to be on the safe site.

    The maximum length default is 8192 bytes and can be read and set via [get|set]MaxStrnlen(..) per class.

    FIXME: strnlen(..) using EOS byte non-functional for non 8-bit codecs like UTF-8, US-ASCII.
           This is due to e.g. UTF-16 doesn't use an EOS byte, but interprets it as part of a code point.

  - TODO: Perhaps a few more unit tests

  - TODO: Allow plain 'int' to be mapped in structs IFF their size is same for all MachineDescriptions used.
          Currently this is the case -> 4 bytes like int32_t.
Reply | Threaded
Open this post in threaded view
|

Re: GlueGen Struct Generator Update *API Change* .. etc

Sven Gothel
Administrator
I like to add and was allowed to mention, that this feature work has been funded by Ankama. Marking this the 2nd round after the MacOS and HiDPI revision as of late. Thank you!

(We had one more last Dec/Jan for MacOS aarch64, but got no allowance to mention 'em)

If more small to medium sized companies using our work would join in the funding rounds, we surely would not have any issues ..

Reply | Threaded
Open this post in threaded view
|

Re: GlueGen Struct Generator Update *API Change* .. etc

Sven Gothel
Administrator
In reply to this post by Sven Gothel
Discussion-1: In cases where the memory is being replaced and the complete 'array' not being written due to 'destPos + length < elemCount', shouldn't the old content be copied into the new memory area?

See example below.

I would say yes ...

+++

Discussion-2: In 'const type*' cases where the memory is always being replaced even at 'destPos + length < elemCount', the elemCount can shrink. This is different than for 'type*' cases, where the setter never shrinks the elemCount.
Shall both variants behave the same, i.e. the 'const type*' case not shrink?

See example below.

I would say yes ...

Additionally, perhaps shall the setter contain a flag allowing the elemCount to be shrinked?
This also would help aligning the setter for both cases.

+++

See setter code for a 'const int32_t*', always replacing whole memory.
- Note: If destPos > 0, the new memory only has zero'ed bytes
- Note: elemCount shrinks if destPos + length < oldElemCount, contrary to non-const setter below

  /** Setter for native field <code>constInt32PointerVariaLen</code>: CType[(PointerType) 'int32_t *' -> (const int32_t) * , size [fixed false, lnx64 8], [const[false], pointer*1]], with initial array length of <code>0</code> */
  public final TK_Field setConstInt32PointerVariaLen(int[] src, final int srcPos, final int destPos, final int length) {
    if( 0 > srcPos || 0 > destPos || 0 > length || srcPos + length > src.length ) { throw new IndexOutOfBoundsException("src[pos "+srcPos+", length "+src.length+"], destPos "+destPos+", length "+length); }
    final int newElemCount = destPos + length;
    final ElementBuffer eb = ElementBuffer.allocateDirect(4, newElemCount);
    ( ( IntBuffer)(eb.getByteBuffer().asIntBuffer().position(destPos) ) ).put(src, srcPos, length).rewind();
    eb.storeDirectAddress(getBuffer(), constInt32PointerVariaLen_offset[mdIdx]);
    _ebConstInt32PointerVariaLen = eb;
    setConstInt32PointerVariaLenElemCount( newElemCount );
    return this;
  }
See setter code for a 'int32_t*', only replacing whole memory if 'destPos + length > elemCount'.
- Note: If destPos > 0 and destPos + length > elemCount, the new memory only has zero'ed bytes
- Note: elemCount never shrinks if destPos + length < oldElemCount, contrary to const setter above
- Note: To shrink the array, one would need to call 'releaseVariaInt32PointerVariaLen()'

  /** Setter for native field <code>variaInt32PointerVariaLen</code>: CType[(PointerType) 'int32_t *' -> (int32_t) * , size [fixed false, lnx64 8], [const[false], pointer*1]], with initial array length of <code>0</code> */
  public final TK_Field setVariaInt32PointerVariaLen(int[] src, final int srcPos, final int destPos, final int length) {
    if( 0 > srcPos || 0 > destPos || 0 > length || srcPos + length > src.length ) { throw new IndexOutOfBoundsException("src[pos "+srcPos+", length "+src.length+"], destPos "+destPos+", length "+length); }
    final int elemCount = getVariaInt32PointerVariaLenElemCount();
    if( destPos + length <= elemCount ) {
      final ElementBuffer eb = ElementBuffer.derefPointer(4, elemCount, getBuffer(), variaInt32PointerVariaLen_offset[mdIdx]);
      ( ( IntBuffer)(eb.getByteBuffer().asIntBuffer().position(destPos) ) ).put(src, srcPos, length).rewind();
    } else {
      final int newElemCount = destPos + length;
      final ElementBuffer eb = ElementBuffer.allocateDirect(4, newElemCount);
      ( ( IntBuffer)(eb.getByteBuffer().asIntBuffer().position(destPos) ) ).put(src, srcPos, length).rewind();
      eb.storeDirectAddress(getBuffer(), variaInt32PointerVariaLen_offset[mdIdx]);
      _ebVariaInt32PointerVariaLen = eb;
      setVariaInt32PointerVariaLenElemCount( newElemCount );
    }
    return this;
  }
Reply | Threaded
Open this post in threaded view
|

Re: GlueGen Struct Generator Update *API Change* .. etc

Sven Gothel
Administrator
If my answer would be taken, we would have:

- Copy old -> new non-covered data when allocating new memory
- Adding a 'allowShrink' to the setter

???
Reply | Threaded
Open this post in threaded view
|

Re: GlueGen Struct Generator Update *API Change* .. etc

Sven Gothel
Administrator
Sven Gothel wrote
If my answer would be taken, we would have:

- Copy old -> new non-covered data when allocating new memory
- Adding a 'allowShrink' to the setter

???
Could name it 'boolean subset', i.e. if true, the set*(..) is meant to only set a subset of the 'array',
hence no shrink will occur. Otherwise, if false, the call is meant to set the whole 'array' and may shrink it.

So if 'subset == true', the 'array' should also not grow - this in contrast to a flag named 'allowShrink'.

Which variant tasted better?
I like to avoid generating another function like setSub*(..) of course.
Reply | Threaded
Open this post in threaded view
|

Re: GlueGen Struct Generator Update *API Change* .. etc

Sven Gothel
Administrator
OK, the setter will be revised:

A) int32_t val[10] -> setter for a subset change only, as-is with args[src, srcPos, destPos, length]
                     - no new memory
                     - no special action to be performed

B) int32_t* val -> setter with toggle 'subset' with args[subset, src, srcPos, destPos, length],
                     - may replace memory where it also may copy 'destPos' gap from old -> new

C) const int32_t* val -> setter to replace referenced memory only args[src, srcPos, length]
                     - always replaces memory, no copy of a gap (no destPos)

Hence:
- case (A) stays as is
- case (B) will be revised and argument 'boolean subset' added
- case (C) will be revised (simplified) and argument 'destPos' removed

Acceptable?
Will do a code of this, also adding a generated API doc header for the setter and getter.
Reply | Threaded
Open this post in threaded view
|

Re: GlueGen Struct Generator Update *API Change* .. etc

Sven Gothel
Administrator
Reply | Threaded
Open this post in threaded view
|

Re: GlueGen Struct Generator Update *API Change* .. etc

Sven Gothel
Administrator
Working pointer-pointer and function-pointer within structs:

https://jogamp.org/gluegen/doc/GlueGen_Mapping.html#struct-pointer-pointer-support

.. and the git logs of course.
https://jogamp.org/cgit/gluegen.git/log/

I pushed most logic down to Type* and Element/PointerBuffer,
so the delta within JavaEmitter is almost neutral.

Left todo: Complete the Java Callback support
Reply | Threaded
Open this post in threaded view
|

Re: GlueGen Struct Generator Update *API Change* .. etc

Sven Gothel
Administrator
This work was joined by Mathieu Féry (Ankama), providing ideas, patches and use-cases.

I was also able to merge his GlueGen patch series, enhancing usability.