It is very difficult for a cracker to interpret what your bytecode is doing. However, please consider the tips provided here for additional security.
You can make a SCRAMBLECODE function, which tests the DLL is unchanged, simply by importing the DLL bytes to a blob and creating some sort of a hash to compare against a constant.
However, a sophisticated attack might be able to perform changes to the loaded image or to compromise resources used by your DLL, which goes undetected. The implications are, you can't be absolutely certain that a particular function returns the correct values or executes as expected.
This notion leads us to the following tips:
The cracker will expect the programmer to be writing well structured algorithms. This means that picking up a few pieces of information here and there (even without understanding the encrypted memory content) may indicate to the cracker what is happening in between, and it may assist the cracker in pinpointing where to focus the attention.
If at all possible try to create variations to the execution flow. Random changing functionality is difficult to interpret and exploit, especially with regards to dynamic attacks implemented as a conditional crack. Not only does it impact the logical execution order, but it may also indirectly affect the encryption and storage of the protected variables in memory.
Here is a few tips:
If a secret algorithm generates a very noticeable temporary value as part of a confidential operation, try to incorporate a randomized initial base value which serves as an offset for the operations, perhaps ended by a final elimination of the offset.
The presented tips add more complexity and consequently more code to prevent dynamic attacks. At the very least the added instructions may create a larger (and preferable randomized) distance between a potential trigger state and its exploitation point, thus making a dynamic attack harder to implement.
TIP: Consider using Group Lines to switch random functionality on/off while debugging and testing your code.
Usually it is not advisable to create your own proprietary encryption or hash algorithms. The more renown algorithms are usually much more secure.
However, it can be advisable to use the renown algorithms in your host application combined with proprietary algorithms in your bytecode - the reason being - if a cracker gets the notion that some bytecode implements a well known algorithm, then a lot of the bytecode is revealed even without understanding the bytecode in detail.
Implementing a proprietary imaginative algorithm will be much more difficult to interpret and may even exist in different versions (e.g. using multiple simple hash algorithms) selected at random.