How can I get a bullet prefab to travel through an invisible wall

Hey, so I am new to unity and right now I am using a simple AI type code for an enemy, I am making a Mario like game, that bounces between two set points(walls) that I have set up with box colliders. I then have to make the colliders trigger to get my character to move through and I know that, however with how I have the bullet code set up the bullet will go right to the colliders and destroy itself, here is the code for the bullet itself

public class Bullet : MonoBehaviour
{
    public LayerMask immaterial;
    public float speed = 20f;
    public int damage = 40;
    public Rigidbody2D rb;
    private GameObject LeftWall;
    // Start is called before the first frame update
    void Start()
    {
        // Velocity of the bullet, right instead of forward for 2D.
        rb.velocity = transform.right * speed;                          
    }    
    // If the bullet hits anything destroy the bullet, if it hits enemy then enemy takes damage.
    private void OnTriggerEnter2D(Collider2D hitInfo)
    {
        Enemy enemy = hitInfo.GetComponent<Enemy>();
        if (enemy!= null)
        {
            enemy.TakeDamage(damage);
        }
       
        Destroy(gameObject);
    }
}

I used the Immaterial to try and fix what I thought would have been a layer issue. It was not. Any insight as to how I could work around this or if I would have to change to a raycast setup for my bullet or what I am supposed to do as I have been looking for about three hours total on this with every search giving me the exact opposite of what I need.

@unity_19borkcdr
At the start of your OnTriggerEnter2D function, add this:

if(hitInfo.isTrigger)
        {
            return;
        }

This will make the bullet check if what it touched was a trigger, and if it is it’ll ignore it.

Disable collision between the wall and bullet

using UnityEngine;
using System.Collections;

public class ExampleClass : MonoBehaviour
{
    public Transform bulletPrefab;

    void Start()
    {
        Transform bullet = Instantiate(bulletPrefab) as Transform;
        Physics.IgnoreCollision(bullet.GetComponent<Collider>(), GetComponent<Collider>());
    }
}