Slope stability has always been a concern for many scholars. Compared with other natural or artificial slopes, open-pit coal mine slopes characterized by high height, long strike, and generally unmaintained (Shu 2009; Wu et al. 2022). Therefore, once a landslide occurs, it is often characterized by a large sliding area, a large volume of the slide and a large surface drop (Li et al. 2022). Rainfall infiltration is one of the main factors causing slope instability. When surface water infiltration encounters water-sensitive soils or soft rocks, the strength of water-sensitive soils or soft rocks will be significantly reduced and accompanied by a significant increase in deformation, which makes the slope equilibrium state disrupted and then causes slope disaster accidents such as landslides (Wang et al. 2018; Ye et al. 2015; Long et al. 2019). Mudstone, as a widely distributed water-sensitive soft rock in China (He et al. 2021; Zhu et al. 2021), endangers the safe production and operation of open-pit mines in various regions(Zhao et al. 2020), and is one of the important challenges for mining engineering and geotechnical engineering research.

Current research on water-sensitive soils and soft rocks focuses on three aspects: deformation, strength and pore structure. The deformation characteristics of water-sensitive soils and soft rocks are mainly divided into collapsibility and expansibility. Among them, collapsible water-sensitive soils and soft rocks are mainly represented by wet-collapsible loess, while expansive soils and soft rocks mainly include expansive mudstone and clay. As the large deformation characteristics of water-sensitive soils and soft rocks brought great harm to engineering construction, scholars have carried out a lot of research on the deformation characteristics of water-sensitive soils and soft rocks. Shao et al. (2018) used a WG tripe triple high pressure consolidator to perform graded wetting tests on remodeled loess with different dry densities, and found that the critical collapse saturation degree of remodeled loess increased with the increasing initial dry density and decreased as the vertical pressure increased. Xie et al. (2018) conducted single oedometer-collapse tests on Malan, upper Lishi and lower Lishi loess soils, and observed that the collapse potential-pressure curves may present bimodal characteristics. Kong et al. (2018) conducted swelling deformation tests on yellow–brown and magenta mudstones from a representative landslide site in the Yanji section of the Jilin-Hunchun high-speed railway using a consolidation instrument, and found that the swelling strain of both mudstones increased with decreasing initial water content and vertical pressure. Dai et al. (2021) proposed a decoupling analysis method for the seepage and swelling of red-bed mudstone, predicting the process of water infiltration through the fissures into the mudstone and the temporal and spatial variation of swelling caused by the slow water absorption of the mudstone.

The weakening of the strength of water-sensitive soils and soft rocks when encountering water is very significant, which seriously affects the stability of tunnels, slopes, and other engineering projects. Many scholars have studied the changes in the strength of water-sensitive soils and soft rocks when encountering water. Xu et al. (2009) observed that the strength of mudstone weakened by water at different stress stages by experiments. Liu and Li (2012) carried out direct shear tests on natural and saturated schist and concluded that the cohesion, internal friction angle, residual cohesion and residual internal friction angle of saturated schist were reduced by 36.7%, 4.6%, 66.7% and 11.3%, respectively, compared to the natural state. Li et al. (2019) pointed out that the uniaxial compressive strength and elastic modulus of mudstone gradually decreased and stabilized with the increase of clay mineral content. Yao et al. (2021) found that the shear strength, cohesion and internal friction angle of mudstone samples decreased exponentially, negative exponentially and linearly, respectively, with increasing water content.

The pore structure is one of the most important factors determining the mechanical behavior and physical properties of soils and soft rocks (Enrique and Paul 2008; Zhang et al. 2023). Sun et al. (2020) conducted environmental scanning electron microscope (ESEM) and MIP tests on bentonite samples with two different initial dry densities at various suctions along wetting and drying paths, and pointed out that slight increase in fractal dimension of bentonite with increasing suction and with increasing dry density. Wang et al. (2020a, b) observed that loess collapse is mainly caused by the destruction of large ang medium pores, where the destruction of medium pores contributed more significantly to the loess collapse. Wei et al. (2020) pointed out that the swelling and dispersion of clay mineral aggregations especially the degraded illite and illite/smectite mixed layer, and the loss of high suction initiate structural failure under loading and wetting conditions. Ge et al. (2021) confirmed the significant differences in pore size distribution and structural arrangement between samples with the same vertical effective stress following different loading and saturation paths by one-dimensional (1D) compression tests, MIP tests and scanning electron microscopy (SEM) tests on compacted loess from Xi’an.

In summary, scholars have conducted a large number of studies on deformation, strength and pore structure of water-sensitive soils and soft rocks, but there are few studies on SWCC of water-sensitive mudstones. SWCC is used to describe relationship between matrix suction and volumetric water content of unsaturated soils and to predict the permeability of unsaturated soils (Kristo et al. 2019). In addition, studies on the pore structure of water-sensitive soils and soft rocks have mostly focused on research objects such as loess and swelling soils, and there are fewer studies on the changes in the pore structure of mudstone in contact with water.

On October 1, 2020, a landslide occurred in the upper 4 steps of the slope in the Shengli No. 1 open-pit coal mine due to continuous rainfall, with a sliding volume of about 220,000 m³ and a surface subsidence of about 3 m, damaging the transportation channel and affecting normal production. In this study, the working slope containing water-sensitive mudstone of Shengli No.1 open-pit coal mine was used as the engineering background. The shear strength parameters of saturated–unsaturated mudstone were obtained by triaxial tests and filter paper method. The pore structure of mudstone with different water contents was studied through MIP testing. The stability of the working slope under different rainfall conditions was studied by numerical simulation method. This study is of great significance for comprehensively understanding the water-sensitive characteristics of mudstone, revealing the impact of water-sensitive mudstone on slope stability, and for disaster prevention and control of slopes containing mudstone weak layers.

2.1 Experimental materials

The mudstone in this study was taken from the mudstone step in the working slope of Shengli No.1 open-pit coal mine, as shown in Fig. 1. The Shengli No.1 open-pit coal mine is located in the southwest of Shengli Coalfield in Inner Mongolia Autonomous Region, China. It is 6 km south of the northern suburbs of Xilinhot City. The geographical coordinates of the mining area are 115° 56′ 51″–116° 01′ 50″ east longitude and 43° 58′ 16″–44° 03′ 10″ north latitude.

Sampling location of mudstone

The mudstone sampled is dark gray in color and is prone to disintegration and expansion when exposed to water. The mudstone was tested for water content, density, free expansion, liquid limit, plastic limit and saturated permeability coefficient. The basic physical properties of the mudstone are shown in Table 1.

As can be seen from Table 1, the mudstone is weakly expansive, with a large plasticity index and poor permeability. X-ray whole rock diffraction and clay separation diffraction tests were performed on the mudstone. The scanning speed was 2°/min and the sampling step width was 0.02°. The mineral composition content of the mudstone is shown in Tables 2 and 3.

According to Tables 2 and 3, the mudstone is mainly composed of clay minerals and quartz. The proportion of clay minerals reached 47.5%. The main clay minerals are illite/smectite mixed layer, kaolinite and illite. The illite/smectite mixed layer is a water-sensitive mineral that accounts for a relatively high proportion in the mudstone. Therefore, the mudstone is highly water-sensitive and is prone to disintegration and expansion when exposed to water.

2.2 Shear strength test of saturated–unsaturated mudstone

The saturation of rock and soil mass at different positions in the slope often varies due to the influence of groundwater and rainfall infiltration. For the same rock and soil mass, there are significant differences in shear strength under different saturation. The shear strength of rock and soil mass directly determines the stability of the slope, so it is necessary to study the shear strength of rock and soil mass with different saturation.

2.2.1 Triaxial tests

The gas in the pores inside the saturated mudstone is completely extruded, and the total stress consists of the effective stress borne by the mudstone skeleton and the pore water pressure borne by the pore water. According to the Mohr Coulomb strength theory and the Terzaghi principle of effective stress, the shear strength of saturated mudstone is:

where τ_f is shear strength of the mudstone, c′ is effective cohesion of the mudstone, σ is normal total stress of the mudstone, φ′ is effective internal friction angle of the mudstone, and u_w pore water pressure in the mudstone.

The consolidated undrained (CU) triaxial tests for saturated remodeled mudstone at different confining pressures (σ₃) was performed by the GDS triaxial test system (Fig. 2) to measure the effective cohesion and effective internal friction angle of the mudstone. The confining pressures (σ₃) are 50 kPa, 100 kPa, 150 kPa, 200 kPa, 250 kPa and 300 kPa respectively.

GDS triaxial test system

2.2.2 Measurement of matrix suction by filter paper method

Unsaturated mudstone contains pores that are not filled with water and is usually considered to be a four-phase system of solid phase (mudstone particles), liquid phase (pore water), gas phase (pore gas), and air–water interface (shrinkage film) (Fredlund and Morgenstern 1977). Shrinkage film affects the mechanical properties of mudstone by pulling the mudstone particles together through surface tension (Kristo et al. 2019). The stress state of unsaturated mudstone is represented by two independent stress state variables, net normal stress (σ − u_a) and matric suction (u_a − u_w) (Fredlund and Morgenstern 1977; Matyas and Radhakrishna 1968). The shear strength of unsaturated mudstone based on the Mohr–Coulomb strength theory and the Terzaghi principle of effective stress can be expressed as (Bishop and Blight 1963):

where u_a is pore-air pressure in the mudstone, and χ is the parameter related to the saturation of the mudstone. When the mudstone is saturated, χ = 1, and when the mudstone is dry, χ = 0.

The shear strength of unsaturated mudstone based on the dual stress variable theory can also be expressed as (Fredlund et al. 1978):

where φ′ is the internal friction angle corresponding to the net normal stress (σ − u_a), and φ^b is the internal friction angle corresponding to the matrix suction (u_a − u_w).

The matrix suction of the mudstone was measured and the SWCC of the mudstone was plotted by the filter paper method (as shown in Fig. 3) to obtain the relationship between matrix suction and volumetric water content of the mudstone. The filter paper used in the test was "Shuangquan" No. 203, and the equation of the matrix suction curve (Wang et al. 2003) was as follows:

where h_m is matrix suction of mudstone and ω_fp is mass water content of filter paper.

Matrix suction measurement test

2.3 MIP tests

MIP tests are based on capillary law of non-wetting liquid (e. g. mercury) to measure the pore diameter of rock and soil mass, as shown in Eq. (5) (Washburn 1921).

where D is pore diameter of rock and soil mass, σ_Hg is surface tension of mercury, θ_Hg is contact angle of mercury with the rock and soil mass, and P is intrusion pressure. In this study, the σ_Hg = 0.484 N/m and θ_Hg = 130°.

The effect of different water contents on the pore structure of mudstone was tested by MIP. The dry density of the prepared mudstone samples is 1.4 g/cm³, and the mass water content is 30% and 48% (saturated water content), respectively. The prepared mudstone samples were dried using the freeze-drying method, and the freeze-drying time was 24 h.

To quantitatively describe the irregularity and surface roughness of the micro pore structure of rock and soil mass, the fractal theory has been introduced into the study of the microstructure of rock and soil mass. The fractal dimension value of the pore structure of the rock and soil mass should be between 2 and 3. The larger the fractal dimension value is, the rougher the pore surface is, and the more complex the pore structure is Zhang and Bing (2015). In this study, the thermodynamic model was used to quantitatively describe the complexity of the pore structure and the roughness of the pore surface in mudstones with different water contents. The thermodynamic model is based on the thermodynamic relationship of porous media during mercury intrusion testing. It is considering the balance between the surface energy increased by mercury intrusion and the work done by external forces on mercury (Zhang and Li 1995):

Substituting Eqs. (7) and (8) into Eq. (6) leads to:

where W_n is accumulated surface energy, Q_n is a function of pore radius and pore volume, C and C’ are constants, i is the ith intrusion step, n is the total intrusion number, \(\overline{p}_{i}\) is the average intruded mercury pressure for ith intrusion, ΔV is the intruded mercury volume for ith intrusion, r is pore radius, and D_z is fractal dimension calculated from the thermodynamic model.

2.4 Numerical simulation

In this study, the working slope of Shengli No.1 open-pit coal mine was taken as the engineering background. The working slope has a height of 220 m and contains four layers of coal seams. The typical profile of the working slope is shown in Fig. 4. The physical and mechanical parameters of the rock strata in the slope are shown in Table 4, where the effective cohesion and effective internal friction angle of the mudstone are taken according to the results of the triaxial tests in Sect. 3.1.

Typical profile of the working slope in Shengli No.1 open-pit coal mine

2.4.1 Numerical model and simulation schemes

In this study, COMSOL Multiphysics software was used to numerically analyze the stability of the working slope under different rainfall conditions. COMSOL Multiphysics is a software that can simulate and calculate multi-physics field. The two-dimensional numerical model of the working slope was established according to Fig. 4 and imported into COMSOL Multiphysics software. The coupling of the stress field and seepage field of the slope model was realized and numerically calculated with the help of the solid mechanics interface and Darcy's law interface in the software.

The numerical model of the working slope is 1840 m long and 270 m high. 26,738 domain units and 10,455 boundary units were divided in COMSOL Multiphysics software for meshing the numerical model of the working slope, as shown in Fig. 5.

Meshing of working slope numerical model

Xilinhot is a semi-arid grassland climate zone with cold winter and hot summer. The annual average precipitation is 294.74 mm. The rainy season is in June, July, and August of each year. The precipitation in the rainy season accounts for about 70% of the annual precipitation. The annual maximum precipitation is 511.0 mm, the monthly maximum precipitation is 281.8 mm, and the daily maximum precipitation is 86.8 mm. To study the stability changes of the working slope under different rainfall conditions, and considering the limited influence of a single rainfall on slope stability (Pan et al. 2020; Wang et al. 2020a, b; Tang et al. 2018; Cho 2016; Qian et al. 2021), we designed different rainfall simulation schemes for the combination of antecedent rainfall and major rainfall according to the classification standard of rainfall intensity and the historical rainfall conditions in the Xilinhot area. Among them, the antecedent rainfall is all with a rainfall intensity of 10 mm/d and a rainfall duration of 5 days. The time interval between the antecedent rainfall and the major rainfall is 5 days. The major rainfall simulation schemes are shown in Table 5.

2.4.2 Numerical model initial conditions

In this study, the safety factors of the working slope under different rainfall conditions were calculated by the strength reduction method. The shear strength parameters of the rock and soil were reduced by a custom program, and the results of the reduced calculation were substituted into the working slope numerical model for calculation through parameter scanning. The infiltration process of rainwater in the working slope was described using the Richards equation (Richards 1931):

where k_ux(h), k_uy(h) and k_uz(h) are the unsaturated permeability coefficients of the rock and soil in x, y and z, h is the pressure head, H is the total head, S is the rainfall flow, and t is the time. In this study, we assumed that the rock and soil are isotropic, that is, k_ux(h) = k_uy(h) = k_uz(h) = k_u(h).

According to the SWCC of unsaturated rock and soil, if C = ∂θ/∂h, then Eq. (10) becomes:

where C is the specific water capacity and is the slope of the SWCC. The equation for the specific water capacity (C) is defined in the Van Genuchten model:

where Se is the effective saturation, which is given by the equation:

In addition, the unsaturated permeability coefficients of the rock and soil (k_u(h)) in Eq. (10) can be given by the equation:

where k_r is relative permeability coefficient.

The Van Genuchten model defines the relative permeability coefficient (k_r) as:

The a, m, and n in Eqs. (12), (13), and (15) are the fitting parameters of the Van Genuchten model, which were fitted based on the test results in Sect. 3.2.

The displacement boundary and initial water pressure boundary of the slope numerical model were set as follows:

(1) Set a fixed constraint at the undersurface of the slope numerical model (y = 0), specifying that the displacement at the undersurface of the model is 0 in all directions.

(2) Set displacement constraints on both sides of the slope numerical model (x = 0, x = 1840), specifying that the x-direction displacement on both sides of the model is 0.

(3) Set the water table in the slope numerical model as follows:

(4) Set water pressure boundaries on both sides of the slope numerical model. The water pressure boundary on the left side of the model (x = 0, 0 ≤ y ≤ 180) is [9800 × (180—y)] Pa, and the water pressure boundary on the right side of the model (x = 1840, 0 ≤ y ≤ 45) is [9800 × (45- y)] Pa.

3.1 Effective shear strength of the mudstone

The failure of the saturated remolded mudstone sample after the consolidated undrained (CU) triaxial tests is shown in Fig. 6. The saturated remolded mudstone samples under different confining pressures (σ₃) all exhibited significant shear expansion failure after CU triaxial tests.

Failure of mudstone samples

The stress–strain curves of saturated remolded mudstone samples under different confining pressures are shown in Fig. 7a. The stress–strain curves of saturated remolded mudstone samples can be roughly divided into three stages:

1. (1)

   When the axial strain ε < 1%, the stress–strain curves are approximately linear, and the saturated remodeled mudstone samples are in an elastic deformation state.

2. (2)

   When the axial strain is 1% < ε < 3%, the stress–strain curves become nonlinear curves, and the plastic proportion of the saturated remolded mudstone samples increases. When confining pressure (σ₃) is 100 kPa and 150 kPa, the rate of increase of deviatoric stress (q) with the increase axial strain ε gradually slows down, and when the deviator stress (q) increases to a peak, it begins to decrease. When the confining pressure (σ₃) is 50 kPa, 200 kPa, 250 kPa and 300 kPa, the rate of increase of deviatoric stress (q) gradually decreases but no peak occurs.

3. (3)

   When axial strain ε > 3%, the deviator stress (q) continues to slowly increase with the increase of axial strain (ε).

Triaxial test results under different confining pressures

As shown in Fig. 7b, the peak deviatoric stress (q) of the saturated remolded mudstone samples exhibits a linear relationship with the variation of confining pressure.

The variation of pore water pressure (u_w) with axial strain ε for saturated remodeled mudstone samples under different surrounding pressures is shown in Fig. 7c. The pore water pressure-strain curves of saturated remolded mudstone samples can also be roughly divided into three stages:

1. (1)

   When the axial strain ε < 1%, the pore water pressure (u_w) increases linearly with the increase of axial strain (ε).

2. (2)

   When the axial strain is 1% < ε < 3%, the increase rate of pore water pressure (u_w) gradually slows down with the increase of axial strain (ε).

3. (3)

   When axial strain ε > 3%, the pore water pressure (u_w) tends to be stable with the increase of axial strain (ε).

As shown in Fig. 7d, the peak pore water pressure of the saturated remolded mudstone samples shows a linear relationship with the variation of confining pressure.

The Mohr–Coulomb failure envelope of the mudstone is plotted according to the Mohr–Coulomb strength theory and Terzaghi effective stress principle, as shown in Fig. 8. The intercept of the Mohr–Coulomb failure envelope is the effective cohesion c′ of the mudstone, and the slope is the tangent value of the effective internal friction angle φ′ of the mudstone. From Fig. 8, it can be seen that the effective cohesion (c′) is 20.24 kPa, and the effective internal friction angle (φ′) is 17°.

Mohr–Coulomb failure envelope of the mudstone

3.2 SWCC of the mudstone

In this study, Logistic curve was selected to fit the filter paper test data after comparing numerous curves in Origin software. The Logistic curve equation is as follows:

where A₁, A₂, x₀ and p are the fitting parameters.

The matrix suction is 0 when the mudstone is saturated. However, there is no 0 point in the logarithmic axis, therefore the matrix suction when the mudstone is saturated is taken as 1 kPa. The curve fitted to the test data based on the logistic curve equation is shown in Fig. 9a. The fitting parameters A₁ = 68.86114, A₂ = 16.97179, x₀ = 54.70808 and p = 1.40178. The fitting degree R² = 0.9805 indicates a good fitting effect.

SWCC of the mudstone

The SWCC can be divided into three zones as the volume water content of the mudstone decreases: boundary effect zone, transition zone and residual zone. By solving the coordinates of feature points A and B in Fig. 9a, it was found that the air entry value (AEV) of the mudstone is 13.57 kPa, the saturated volume water content (θ_s) is 67.2%, the residual volume water content (θ_r) is 20.79%, and the residual matrix suction (ψ_r) is 193.97 kPa.

When the matrix suction is less than 13.57 kPa, the mudstone is in a saturated or near-saturated state, and its internal pores are filled with water and the pore water is interconnected. When the matrix suction gradually increases to more than 13.57 kPa, air gradually enters the internal pores of the mudstone. As the further increase of the matrix suction, the water and air in the internal pores of the mudstone are in an unconnected state, and the matrix suction generated by the shrinkage film significantly increases with the decrease of the volume water content of the mudstone. When the matrix suction increases to 193.97 kPa, the air in the internal pores of mudstone is interconnected, and only a small portion of water exists in discontinuous small pores. At this time, the mudstone is close to a dry state, and the change in matrix suction has little effect on the volume water content of mudstone.

It is worth noting that although the Logistic curve equation fits the test data well, the left curve of the boundary effect zone of the SWCC is higher than the saturated volume water content of the mudstone, and the fitting parameters have no physical significance. Further improvement is still needed for this fitting curve.

Many empirical models for predicting SWCC by fitting fewer test data have been proposed by many scholars on the basis of a large number of experiments, due to the high cost, complex operation, or long cycle, and high workload of matrix suction measurement methods. Among them, the more classic models include the Gardner model (Gardner 1958), Van Genuchten model (Van Genuchten 1980), and Fredlund & Xing model (Fredlund and Xing 1994), as shown in Table 6. Here, a, b, m and n are the fitting parameters of each SWCC model.

The SWCCs of mudstone based on three classical SWCC models fitted to the test data are shown in Fig. 9b, and the model fitting parameters and degree are shown in Table 7.

As shown in Fig. 9b and Table 7, all three typical SWCC models are well fitted, and the fitting degree reach above 0.97. Compared with the Logistic curve, all three typical SWCCs pass the saturated point of the mudstone. The Gardner and Van Genuchten models fit the boundary effect zone and transition zone of the SWCC better, but the residual zone deviates from the actual data. The Fredlund & Xing model fits the boundary effect zone, transition zone, and residual zone of the SWCC well. Therefore, the Fredlund & Xing model is more suitable for fitting and predicting the SWCC of the mudstone.

3.3 Pore size distributions of the mudstones with different water content

Figure 10 shows the pore size distribution (PSD) curves of the mudstone samples with different water contents. It can be seen from Fig. 10a that the total pore volume of the mudstone samples with 30% mass water content is larger than that of the mudstone samples with 48% mass water content. The shape of cumulative pore volume curves was similar, and the slope of the curves changes at pore diameters of approximately 0.1 μm, 1.5 μm and 25 μm. To visually analyze the pore content of mudstone samples with different water contents, the pores of mudstone are divided into micropores (< 0.1 μm), small pores (0.1–1.5 μm), mesopores (1.5–25 μm), and macropores (> 25 μm) according to their diameters.

PSDs of mudstone samples with different water contents

Figure 10b shows the pore size density functions of mudstone samples with different water contents are bimodal, in which the first peak is in the small pores and the second peak is in the mesopores. There is a significant difference in the two peak values of pore size density functions for mudstone samples with different water contents. The pore size density functions of the mudstone samples with different water contents in the micropores and macropores do not differ significantly, indicating that changes in water content have a relatively small impact on micropores and macropores. The most probable pore diameter indicates that the content of pores with that diameter is the highest in mudstone samples. The most probable pore diameter of the mudstone sample with 30% mass water content is 19.10 μm, while the most probable pore diameter of the mudstone sample with 48% mass water content is 6.75 μm. This may be because when the water content increases, the clay minerals in the mudstone expand with water and occupy the space of the mesopores, leading to the shrinkage of the mesopores into small pores.

The fractal dimensions of mudstone samples with different water contents are shown in Fig. 11. According to Eq. (9), the slope of the fitting line in Fig. 11 is the fractal dimension (D_z) of the thermodynamic model. The fractal dimension of mudstone sample with 30% mass water content is 2.87955, while the fractal dimension of mudstone sample with 48% mass water content is 2.84311. This indicates that the change of water content changes the complexity of the pore structure and the roughness of the pore surface inside the mudstone samples. The higher the water content, the simpler the pore structure and the smoother the pore surface inside the mudstone, which causes changes in physical and mechanical properties such as increased permeability and decreased strength of the mudstone.

Fractal dimension of mudstone samples with different water contents

3.4 Stability of working slope under different rainfall conditions

3.4.1 Changes in slope saturation

According to the initial conditions of the slope numerical model in Sect. 2.4.2, the rock strata below the water table are saturated, and the initial effective saturation of the rock strata above the water table is set to 30%, as shown in Fig. 12a.

Effective saturation of rock strata in the working slope under different rainfall conditions

The effective saturation of the rock strata in the working slope under different rainfall conditions is shown in Fig. 12b–i. Due to the overall low permeability of the rock strata in the working slope, there is no hydraulic connection between rainwater and groundwater after rainfall, and only the effective saturation of the surface rock strata in the slope changed. The focus of this study is on the water-sensitive mudstone in the working slope, so the mudstone layer was locally amplified. It can be seen from the local enlargement of the mudstone layer that the effective saturation of the mudstone layer varies under different rainfall conditions. The maximum effective saturation of the mudstone layer surface is 54.80% after the antecedent rainfall. The saturation of the mudstone layer surface decreased slightly after 5 days of the antecedent rainfall, i.e., before the major rainfall, with a maximum effective saturation of 54.60%. When the rainfall intensity is 20 mm/d, the maximum effective saturation of the mudstone layer surface increases to 70.44% after 24 h of rainfall. When the rainfall intensity is 45 mm/d, some areas of the mudstone layer surface are close to saturation after 24 h of rainfall, and the maximum effective saturation increases to 98.24%. When the rainfall intensity is 90 mm/d, after 24 h of rainfall, the mudstone layer surface is fully saturated and the mudstone layer shows obvious saturated and unsaturated transition zones. These results indicate that the greater the rainfall intensity, the greater the effective saturation of the mudstone layer surface when the rainfall duration is the same.

Comparing the simulation results of Scheme II and Scheme IV both with a total rainfall of 45 mm, it can be seen that when the rainfall intensity is 10 mm/d, after 4.5 days of rainfall, the maximum effective saturation of the mudstone layer surface is 93.31%, which is smaller than that in Scheme II. This is due to the fact that the longer the duration of the rainfall, the larger the rainfall infiltration amount when the rainfall total is the same, resulting in a slight decrease in the mudstone layer surface saturation. The mudstone layer surface in Scheme III, V and VI, in which the total amount of rainfall is 90 mm, is fully saturated, and all of them show obvious saturated and unsaturated transition zones. In addition, the simulation results of Scenarios IV and V, in which the rainfall intensity is 10 mm/d, show that when the rainfall intensity is the same, the longer the rainfall duration is, the greater the effective saturation of the mudstone layer surface is.

3.4.2 Changes in slope stability

The safety factors and the critical slip surfaces position of the working slope under different rainfall conditions were obtained based on the calculated non convergence or plastic zone connection as the slope instability criterion. The equivalent plastic strains of the working slope under different rainfall conditions at the time of reduction failure are shown in Fig. 13. The safety factors of the working slope under different rainfall conditions are shown in Fig. 14.

Equivalent plastic strain of the working slope under different rainfall conditions

Safety factors of the working slope under different rainfall conditions

Figure 13 shows that the plastic zones of the working slope at the time of reduction failure under different rainfall conditions appears on the shallow layer of mudstone steps, indicating that the critical slip surfaces of the working slope under different rainfall conditions are all located on the shallow layer of mudstone steps, and the mudstone steps are prone to shallow sliding. Due to the large size of the working slope, only the mudstone weak layer in the upper part of the slope generates plastic strain under rainfall, without forming a plastic zone that runs through the entire slope. Therefore, the influence of the rainfall scheme simulated in this study on the safety factors of the slope is small. However, it can still be used to explore the changes in the stability of a slope containing mudstone weak layers under different rainfall conditions.

As shown in Fig. 14, the safety factors of the working slope decrease under different rainfall conditions. The safety factor of the working slope decreased from 1.180 to 1.170 at the end of the antecedent rainfall. The plastic strain value at the mudstone step 5 days after the end of the antecedent rainfall, i.e., before the major rainfall, is less than the plastic strain value at the mudstone step at the end of the antecedent rainfall, and the safety factor of the working slope increases to 1.171. Comparing the results of each simulation scheme of major rainfall, it can be seen that when the rainfall duration is the same, the greater the rainfall intensity is, the greater the plastic strain in the shallow layer of the mudstone step is, and the smaller the safety factor of the working slope is. When the rainfall intensity is the same, the longer the rainfall duration, the greater the plastic strain in the shallow layer of the mudstone step, and the smaller the safety factor of the working slope. These indicate that the total amount of rainfall is the key factor affecting the stability of the slope. When the total amount of rainfall is the same, the longer the rainfall duration, the smaller the plastic strain in the shallow layer of the mudstone step, and the greater the safety factor of the working slope. It can be seen that the short duration strong rainfall is more threatening to the stability of the slope than the long duration weak rainfall.

The permeability of the rock strata on the working slope of Shengli No.1 open-pit coal mine is poor, and only the water content on the surface of the slope increases under rainfall infiltration. The water-sensitivity of mudstone leads to a significant weakening of its strength and a significant increase in deformation when encountering water, while other rock strata in the slope are non-water-sensitive (in fact, sandy mudstone in the working slope is also weakly water-sensitive. and the weak water-sensitivity of the sandy mudstone is not considered in this study in order to highlight the influence of the weak layer of water-sensitive mudstone on the stability of the slope). After encountering water, the strength will not change significantly. Therefore, the plastic strain occurs only in the mudstone steps after rainfall.

In addition, the fact that rainwater is not contacted with groundwater also results in the absence of a plastic zone that runs through the entire slope. Of course, if the rainfall is large enough and the rainwater keeps infiltrating downward, causing the increase of water content and decrease of shear strength of non-water-sensitive rock strata in a large area of the working slope, as well as the rise of groundwater level, the working slope may form a plastic zone penetrating the slope as a whole and large-scale sliding instability will occur. However, the Xilinhot area is a semi-arid grassland climate with low rainfall in the rainy season and throughout the year, making it unsuitable for large-scale rainfall. Therefore, the key to preventing landslide of the slope containing mudstone weak layer in Shengli No.1 open-pit coal mine is to control the deformation and sliding of the mudstone weak layer.

With the working slope containing water-sensitive mudstone of Shengli No.1 open-pit coal mine, as an example, this study investigated the water sensitive characteristics of mudstone and analyzed the stability of the working slope. Firstly, the shear strength of saturated–unsaturated mudstone was investigated using the GDS triaxial test system and the filter paper method. Secondly, the pore structure of mudstone samples with different water contents was analyzed by the MIP tests combined with the fractal dimension. Finally, the stability of the working slope under different rainfall conditions was numerically calculated. The following conclusions can be drawn from the analysis and discussion of the experiments and numerical calculations:

1. (1)

   The saturated remolded mudstone samples under different confining pressures all exhibited significant shear expansion failure after CU triaxial tests. The curves of deviatoric stress and pore water pressure with the increase of axial strain can be divided into three stages: rapid linear growth, slow growth and stabilization. The peak values of deviatoric stress and pore water pressure are linearly positively correlated with confining pressure. The effective cohesion of the mudstone is 20.24 kPa, and the effective internal friction angle is 17°.

2. (2)

   The SWCC of the mudstone fitted by the Logistic equation shows that the AEV of the mudstone is 13.57 kPa, the saturated volume water content is 67.2%, the residual volume water content is 20.79%, and the residual matrix suction is 193.97 kPa. The Fredlund & Xing model is more suitable for fitting and predicting the SWCC of the mudstone among the three classic SWCC models.

3. (3)

   The total pore content of the mudstone sample with lower water content is greater than that of the mudstone sample with higher water content. The mesopores are more in the mudstone sample with lower water content, while the small pores are more in the mudstone sample with higher water content. This may be because when the water content increases, the clay minerals in the mudstone expand with water and occupy the space of the mesopores, leading to the shrinkage of the mesopores into small pores.

4. (4)

   The fractal dimensions of mudstone samples with different water contents show that the higher the water content, the simpler the pore structure and the smoother the pore surface inside the mudstone, which causes changes in physical and mechanical properties such as increased permeability and decreased strength of the mudstone.

5. (5)

   Under different rainfall conditions, the effective saturation on surface of the working slope increases, the safety factor decreases, and the plastic zone appears on the mudstone steps. The key to preventing landslide of the slope containing mudstone weak layer in Shengli No.1 open-pit coal mine is to control the deformation and sliding of the mudstone weak layer.