keeping non-minimal assets out of code repository

[kortemik]

Having binary blobs in our code repository will create a problem in near future as git always pulls all the diffs on the repo when cloning. This way the binaries will create quite significant size increase for us.

By moving the assets which make most of our binaries to a submodule we can use svn for managing them and therefore code repository will not increase in size.

https://help.github.com/articles/support-for-subversion-clients/

Also for size consideration

https://help.github.com/articles/what-is-my-disk-quota/#file-and-repository-size-limitations

[kortemik]

also the build scripts should be modified so that they notify about submodules as they seem to be a problem for some users

[DanielGibson]

A submodule won't help, because then if you checkout the repo withe submodules you'll download everything, which is exactly what we want to avoid by using SVN.

I'd like to keep the minimum assets (what we have now + maybe some more for the menu) in the main git repo, but use SVN for additional stuff like demo maps with cool textures etc.

So let's not move the assets out of the main repo, but have a second repo for additional non-minimal assets.

[ghost]

Will art assets from my side affect it as well? I'm trying to design to keep file sizes of art assets to a minimum. Maybe a seperate repository for art assets?

[DanielGibson]

yes.

a separate repo for assets is what kortemik is suggesting :-)

My rough idea: we could have the base/ dir like it is now with the absolute basics (editor textures, some simple layout and light textures, everything for the standard menu, entity definitions, ..) and another demo/ dir (or similar) besides that, like a "mod" dir.
demo/ would be from the SVN repo and contain things like your example maps with proper textures, sounds etc.
when starting the game with +set fs_game demo it should use the assets from both base/ and demo/ (this is how mods worked in original doom3)

[ghost]

Agreed that is a better solution.

[BielBdeLuna]

I don't understand it, why separate the base from the demo?

we just need a repository for assets and another for code that is all isn't it?

[ghost]

I'd prefer to keep my changes in a demo folder seperate from the base, so that any of my changes that could be faulty is avoided in the main base. Another reason is that updating from github could take longer as the art assets will grow in size. I'm going to switch from TGA to PNG to reduce file sizes. I think it is best that we decide on a resolution size for textures for optimal performance in the engine.

[BielBdeLuna]

but if we keep the assets in another repository, then the code should npt suffer from any increased sizes on the asset side

besides that, managing a base folder and a mod folder is stranger for the base assets to create a demo, I think we should stick ton a single base folder. there are not assets of first order and assets of the second order, all assets contribute the same to the demo isn't it?

[ghost]

Either way I don't mind which way we do it. My only reason is to avoid issues from constant art asset updates, as I tend to juggle between files to make changes. It's better to ask Kortemik and Caedes.

[BielBdeLuna]

but, there hasn't been any asset updates in a long time isn't it?

[DanielGibson]

regarding assets updates: https://github.com/OpenTechEngine/OpenTechBFG/commits/cegui_layout_upd_assets

There are first order assets and second order assets:
First order assets are things that are elemental to use the engine (like entity definitions and their editor textures) + things that are elemental to execute opentech (like menus).
Second order assets are things that demo what opentech can do like cool looking levels.

[BielBdeLuna]

but wthout a single level the engine is useless so you need the demo to see anything in he engine, or it is just a couple of menus with nothing to play
so no
the demo is capital to the engine

[DanielGibson]

there is a very simple test level using the layout textures.

the demo assets will be part of the project and if we create binary releases we will probably bundle them.

but this is an engine, not a game, so I think it makes sense to keep the main repo small and clean and only have those things in it, that people creating a game with it need to use the engine.

another thing is that while I insist to keep the assets in base/ under WTFPL or similar, demo assets could be under a less liberal license - people creating their own game probably won't use them (and the engine is useful without them).

[BielBdeLuna]

why would someone not use the demo files? I mean, they will create the AI from scratch, they will create all the def files from scratch? obviously not, if they had to do this the engine would be useless because you have to create all from scratch.
the demo doesn't contain anything that shouldn't be capital to have in any game.

[DanielGibson]

the entity definitions and other basic/important things are not part of the demo, but remain in base/, like they're now.

but they will create their own levels and their own textures, their own sounds, ... so that stuff (which happens to be big binary data mostly) ends up in the demo repo.

.def files and ai scripts for monsters are specific to the monsters and thus probably belong in demo, however I could imagine:

1. having simple monsters in base
2. still enforcing WTFPL for their def's and scripts in demo/, so people that want to create a similar monster can enhance that.

[BielBdeLuna]

you can't have simple monsters, it's a good coding effort to have a monster without any base scripting that is really solid, it makes no sense to create a "simple monster"

it makes no sense to create super complex textures in order to show up the capabilities of the engine
most of the demo textures could be dev textures except for those needed textures that are used to show features specific to texture effects, @yetta1 said he was going to make the level of the demo that was meant to show the capabilities with different rooms to show different things.

[DanielGibson]

no idea how complex the script must be, the script isn't a problem anyway, but the monsters model and especially textures can be kept simple/small in filesize, so it doesn't use too much space.

I guess people appreciate pretty textures in an engine demo

(This discussion is pointless and I'll stop here.)

[BielBdeLuna]

no need to have all the textures pretty, just to show up the relevant innovations in a controlled environment

[kortemik]

I agree with you both, although I'd keep base/ in different repo just to ease history filtering and that kind of things if needed.

For my opinion base/ should contain definitions and simple scripts which can be then refined by fs_path. These base/ things could make up a very simple demo of the engine

fs_path demo/ (or other directory, i'd call it example) should then contain the assets for example game. It doesn't need to be big one but that should definitely be in it's own repository.

base/ should kind of be a place for commonly shared things across the games, like character movement animation scripts and basic defs.

These are the reasons that I would like to keep them in their own repository.

Keep up the good work and we will solve these out, git is easy to work with so no need to be scared of this issue :)

[DanielGibson]

maybe the easiest way for now would be to develop additional assets into demo/ and selectively move some of them into base/ when they're ready (so they don't change much anymore and don't blow up the git history).

but either way there's not much point in discussing this right now, we can still do that when we have actual assets and know how big they are etc.

[BielBdeLuna]

the way I'm developing the scripts is meant to be very organized as well as very fragmented so this could be possible: https://github.com/BielBdeLuna/idTechX_GPL_scripts

at the moment is untested code and unfinihsed code

[ghost]

@BielBdeLuna Not sure if I understand correctly, but would this make seperate scripts from modders/devs easier to use with new mods/games in the future?

Just tried to play a D3 mod called Ruiner, but it seems to dislike base folders containing other mods, giving errors about .script files.

[BielBdeLuna]

no, but since they want to separate... I don't know why they intend to separate the base with the demo.

[DanielGibson]

From a quick glance it looks like you use inheritance a lot (which makes sense, I guess).
So one could have the generic AI scripts in base/ and the one specific to a monster (that we have a model for etc) deriving from e.g. ai_bipped in demo/

[BielBdeLuna]

at the moment is ai_bipped, but in the future I will have ai_combatant, and ai_npc

all mosters will be in a /ai/fsm/ folder because in the future I will abandon fsm and use a GOAP system
and so I will have a /ai/goap/ folder too which will need a totally different approximation of AI in the scripting.

before GOAP I will introduce a motionState loop to c++ ( akin to animState, and State ) I already have coded and working but I will do the d3 compatible AI.

the idea is to use several AI systems for several different entities.

[BielBdeLuna]

@yetta1 to me the demo has to be not a game but a single demo map that shows the new capabilities of OTE (new from d3BFG) we can't do a full fledged game, even a small one, the lack of artistic direction, or even quality, would go against the idea of showing the positive qualities of OTE.

[DanielGibson]

Yeah, that's all fine, I'm just saying that it seems reasonable to have the basic AI scripts in base/ and the specializations for actual monsters in demo/.

A map showing the capabilities is certainly a reasonable first step, but having some more maps with different artistic directions (e.g. one that looks cartooney/flat shaded, one with a dark/dungeon-y style, one with a futuristic style, ...) wouldn't hurt either, just to show what results the engine can produce.
(Those were just examples, I'm not saying we need maps in that direction or anything, but if someone feels like creating a cool map for OTE with whatever artistic direction I'm all for including it into the demo. If we have several maps in the demo they don't necessarily need a similar art direction or anything - it's not a real game.)

[BielBdeLuna]

yes but let's first finish the demo map.

for the scripting it makes no sense to have parts of the code there and parts of the code here. it makes for much easier readability to have all the code in a compartmented folder, it's easier to find the correct files if they are set in relevant named folders, and makes for a much more organic way to have the information organized (all AI code is within the /script/ai/ folder, easier to understand and instantaneous to be grasped)

the other important thing here is to have a strong math and subs folders, which will empower future projects

I've also changed the file extensions to *.include (for include files), *.globals (for global definitions) and *.subs for (subroutines files)
the whole idea behind it all is to have an accessible code.

my intention is also are to ask the members of the community to share their special subs so all those subs is definitively GPLv3, or some sort of CC lic, some of them are pretty interesting entities subs and math subs, that will come in very handy in the future.

[ghost]

The current map i'm making for OTE will have a background scene inspired by Blade Runner/Total Recall, the general theme of the map itself is a R&D company developing weapons and robotics. There are different layers to the map where interactive assets can be placed:

Top floor:
Offices
Cantine
Kitchen
Server room
Board room
Bathroom
Locker room

First floor:
Weapons development
Robotics development
Locker room
Reception
Maintenance office

Basement lv1:
Weapons firing range
Storage
Safety equipment storage

Basement lv2:
Robotics battle simulation room
anti-gravity chamber

Basement lv3:
Teleportation experiment room
teleport to another small alien world

That is the general theme i've planned for the map, so there are many opportunities to add feature demos. The BSP layout is done, I just need to finish all the models, textures, sounds, rigging and animations.

[BielBdeLuna]

I would discourage the anti-gravity room, because the unique way to control the gravity at the moment is via cvar fiddling, and this to me is a big nono, we should add a way to gather info on the spot (this would also be useful for having water in the maps: are we in water here or not? are we here in antigravity or not? you know a way to get physical info off the environment at the selected spot )

so we could have gravity amount affect not only entities but particles too,

let's have it but also let's add it to the TODO list of entities and code to add to the code base

[ghost]

There is a sewer part in basement lv2, which can be used for water physics. As for anti_gravity, I was thinking of having a entity that changes the gravity when triggered, like the one in Dead Space when Isaac turns off gravity to get to the other side. I think the Prey SDK contains code for it. For now i'll block access to the gravity room or I can remove it completely.

[BielBdeLuna]

yes, we can see what is going on in prey (but i bet it's pretty complicated) the thing is that we can't use any of the code in the SDK of prey because it's not GPL it's in the EULA of the game which isn't compatible with GPL. we can only watch and recreate it, but not copy it.

Motorsep and I went asking to the Dark Mod team to release their water code s GPL (back at the time they didn't have the code in GPL, it wasn't stated in their files) their code was form the SDK time so their code was not GPL complaint, so now all the water code in the Dark Mod is copyable as long as we release the code GPL.

I attempted to port it to dhewm but I couldn't, I didn't had enough c++ knowledge to port it, so it all ended in nothing. the only thing positive that came out form that effort was a way to have different step sounds for different surfaces (and even water checks) for the monster and player (the player already have this, but the monsters don't) it works and I will port it to OTE someday.

having those checks we could improve on them and make the player constantly check their environment for changes like the one you stated

[ghost]

That's really cool, i'd like to see it in OTE one. Swimming monsters would be cool too. I downloaded dhewm a few days ago, but haven't been able to build a binary for it. Got iodoom3 to work yesterday, but read somewhere that dhewm is far superior to iodoom3.

[BielBdeLuna]

yes we could use Hexen2 as a reference for drowning monsters, although it would be cool to give them the option to swim out of the water, or have both :)

dhewm3 should be the most modern, and with the most advanced features.

[ghost]

I'd say both, it would be cool to have swimming monsters chasing the player, like in Alien 4 resurrection. I still want to play the Hexen Edge of Chaos mod for D3.

[ghost]

Could make them drown when their health reaches a certain threshold and they become to weak to swim up for air.

[BielBdeLuna]

no, they drown when they run out of air, their capability to swim shouldn't influence much positively in their incapability to breath under water XD

[ghost]

I meant for non-aquatic monsters that might dive in after the player. For example they need air to breathe, but while swimming if their health say reaches 5-10% they lose mobility while swimming, causing them to drown once their air runs out.

[BielBdeLuna]

it seems like a very odd limitation for a character, there will be characters unable to swim, like non bippeds, why limit bippeds?
but anyways the functionality will eventually be there, it will be expandable at will.

but there is no credible water in idTechX at the moment, let's not worry about it at the moment.

[kortemik]

https://git-lfs.github.com/

[DanielGibson]

For git-lfs you need to sign up for early access.
We should evaluate it once it's officially released and available for anyone.

Anyway, I created the demo/ repo: https://github.com/OpenTechEngine/demo, please use it for new assets, you can use it via SVN so you don't store the whole history locally.

BTW, idTechX is a strange name, it sounds like either "id Tech 10" or "id Tech x" with x being variable, i.e. one of id tech 1-5
why not just idTech4.5 or idTechBFG? I guess more people could understand that.

[BielBdeLuna]

idTechBFG is speaking of a game based in that engine doom3BFG, it's a very Doom name

idTech4.5 it's strange because it seems like an engine stuck between idTech4 and idTech5, and when we advance the engine to another territory, if we ever do, it might not be that useful to be associated to those engines.

X is not only a roman numeral, X also could mean "next" or "extra" or "number-less" people shouldn't recognize the engine because once powered Doom3BFG but by it's own merits, hence the demo.

while the engine name is actually OTE the tech which at the moment shares with RBdoom3BFG and Doom3BFG vanilla engine needs a name (and I Remember reading somewhere within the engine code that Id called it idTechX, though I haven't found this reference again ) the further we deviate from Doom3BFG vanilla engine the less need for calling idTechX and the more need to actually call it OTE, at the moment the biggest change and the more showy change is the shadow mapping from RBdoom3BFG.

[DanielGibson]

doom3bfg is a mixture of idtech4 and idtech5.

when talking about this particular project, OTE is probably most appropriate, when talking about d3bfg based projects in general like rbdoom3bfg, idtech4.5 seems most appropriate

use whatever you want, but I guess most people would assume that idTechX means "id tech engines in general".